Oct 30 00:28:57.190564 ip-10-0-135-56 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 00:28:57.190715 ip-10-0-135-56 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 00:28:57.190823 ip-10-0-135-56 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 00:28:57.191136 ip-10-0-135-56 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 00:29:07.309501 ip-10-0-135-56 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 00:29:07.309518 ip-10-0-135-56 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot faa9a9218373404e8241d554d5495c82 --
Oct 30 00:31:07.436717 ip-10-0-135-56 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 00:31:07.803923 ip-10-0-135-56 kubenswrapper[2548]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:31:07.803923 ip-10-0-135-56 kubenswrapper[2548]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 00:31:07.803923 ip-10-0-135-56 kubenswrapper[2548]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:31:07.803923 ip-10-0-135-56 kubenswrapper[2548]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:31:07.809302 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809213    2548 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:31:07.809302 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809304    2548 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:31:07.809393 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809309    2548 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:31:07.809393 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809312    2548 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:31:07.809393 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809316    2548 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:31:07.809393 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809319    2548 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:31:07.809393 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809321    2548 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:31:07.809393 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809324    2548 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:31:07.809393 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809327    2548 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:31:07.809393 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809330    2548 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:31:07.809393 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809332    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:31:07.809393 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809335    2548 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:31:07.809393 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809338    2548 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:31:07.809393 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809340    2548 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:31:07.809393 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809344    2548 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:31:07.809393 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809347    2548 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:31:07.809393 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809350    2548 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:31:07.809393 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809367    2548 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:31:07.809393 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809370    2548 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:31:07.809393 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809373    2548 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:31:07.809393 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809375    2548 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:31:07.809393 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809378    2548 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:31:07.809850 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809381    2548 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:31:07.809850 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809383    2548 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:31:07.809850 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809386    2548 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:31:07.809850 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809389    2548 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:31:07.809850 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809397    2548 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:31:07.809850 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809400    2548 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:31:07.809850 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809403    2548 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:31:07.809850 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809407    2548 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:31:07.809850 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809410    2548 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:31:07.809850 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809413    2548 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:31:07.809850 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809416    2548 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:31:07.809850 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809419    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:31:07.809850 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809421    2548 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:31:07.809850 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809424    2548 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:31:07.809850 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809427    2548 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:31:07.809850 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809429    2548 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:31:07.809850 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809432    2548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:31:07.809850 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809434    2548 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:31:07.809850 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809436    2548 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:31:07.809850 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809439    2548 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:31:07.810429 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809441    2548 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:31:07.810429 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809444    2548 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:31:07.810429 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809446    2548 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:31:07.810429 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809448    2548 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:31:07.810429 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809451    2548 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:31:07.810429 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809453    2548 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:31:07.810429 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809458    2548 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:31:07.810429 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809461    2548 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:31:07.810429 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809463    2548 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:31:07.810429 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809466    2548 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:31:07.810429 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809468    2548 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:31:07.810429 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809471    2548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:31:07.810429 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809473    2548 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:31:07.810429 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809477    2548 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:31:07.810429 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809480    2548 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:31:07.810429 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809482    2548 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:31:07.810429 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809485    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:31:07.810429 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809488    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:31:07.810429 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809490    2548 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:31:07.810429 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809493    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:31:07.810890 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809496    2548 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:31:07.810890 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809499    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:31:07.810890 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809501    2548 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:31:07.810890 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809506    2548 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:31:07.810890 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809511    2548 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:31:07.810890 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809514    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:31:07.810890 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809517    2548 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:31:07.810890 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809521    2548 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:31:07.810890 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809524    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:31:07.810890 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809528    2548 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:31:07.810890 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809534    2548 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:31:07.810890 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809538    2548 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:31:07.810890 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809541    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:31:07.810890 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809544    2548 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:31:07.810890 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809547    2548 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:31:07.810890 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809550    2548 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:31:07.810890 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809553    2548 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:31:07.810890 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809555    2548 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:31:07.811383 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809558    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:31:07.811383 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809561    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:31:07.811383 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809563    2548 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:31:07.811383 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809566    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:31:07.811383 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809569    2548 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:31:07.811383 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809572    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:31:07.811383 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809574    2548 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:31:07.811383 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809577    2548 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:31:07.811383 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809580    2548 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:31:07.811383 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809582    2548 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:31:07.811383 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809585    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:31:07.811383 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809588    2548 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:31:07.811383 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809591    2548 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:31:07.811383 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809593    2548 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:31:07.811383 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809596    2548 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:31:07.811383 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809598    2548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:31:07.811383 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809601    2548 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:31:07.811383 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809604    2548 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:31:07.811383 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809606    2548 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:31:07.811383 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809609    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:31:07.811855 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809612    2548 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:31:07.811855 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809615    2548 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:31:07.811855 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809618    2548 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:31:07.811855 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809621    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:31:07.811855 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809624    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:31:07.811855 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.809626    2548 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:31:07.811855 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811155    2548 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:31:07.811855 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811165    2548 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:31:07.811855 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811168    2548 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:31:07.811855 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811171    2548 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:31:07.811855 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811174    2548 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:31:07.811855 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811176    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:31:07.811855 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811179    2548 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:31:07.811855 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811181    2548 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:31:07.811855 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811185    2548 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:31:07.811855 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811188    2548 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:31:07.811855 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811190    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:31:07.811855 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811193    2548 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:31:07.811855 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811195    2548 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:31:07.812302 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811200    2548 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:31:07.812302 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811203    2548 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:31:07.812302 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811206    2548 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:31:07.812302 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811209    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:31:07.812302 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811213    2548 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:31:07.812302 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811217    2548 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:31:07.812302 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811220    2548 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:31:07.812302 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811223    2548 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:31:07.812302 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811225    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:31:07.812302 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811228    2548 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:31:07.812302 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811231    2548 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:31:07.812302 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811234    2548 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:31:07.812302 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811236    2548 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:31:07.812302 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811239    2548 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:31:07.812302 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811241    2548 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:31:07.812302 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811244    2548 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:31:07.812302 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811247    2548 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:31:07.812302 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811250    2548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:31:07.812302 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811253    2548 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:31:07.812761 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811255    2548 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:31:07.812761 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811258    2548 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:31:07.812761 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811261    2548 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:31:07.812761 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811263    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:31:07.812761 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811266    2548 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:31:07.812761 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811268    2548 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:31:07.812761 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811271    2548 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:31:07.812761 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811273    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:31:07.812761 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811276    2548 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:31:07.812761 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811279    2548 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:31:07.812761 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811282    2548 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:31:07.812761 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811284    2548 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:31:07.812761 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811287    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:31:07.812761 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811291    2548 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:31:07.812761 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811293    2548 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:31:07.812761 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811296    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:31:07.812761 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811299    2548 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:31:07.812761 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811301    2548 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:31:07.812761 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811305    2548 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:31:07.812761 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811307    2548 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:31:07.813235 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811310    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:31:07.813235 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811312    2548 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:31:07.813235 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811315    2548 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:31:07.813235 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811318    2548 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:31:07.813235 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811320    2548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:31:07.813235 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811322    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:31:07.813235 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811325    2548 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:31:07.813235 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811328    2548 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:31:07.813235 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811331    2548 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:31:07.813235 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811333    2548 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:31:07.813235 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811336    2548 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:31:07.813235 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811339    2548 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:31:07.813235 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811341    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:31:07.813235 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811344    2548 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:31:07.813235 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811347    2548 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:31:07.813235 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811349    2548 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:31:07.813235 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811366    2548 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:31:07.813235 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811369    2548 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:31:07.813235 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811372    2548 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:31:07.813235 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811374    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:31:07.813732 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811377    2548 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:31:07.813732 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811380    2548 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:31:07.813732 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811384    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:31:07.813732 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811387    2548 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:31:07.813732 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811390    2548 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:31:07.813732 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811392    2548 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:31:07.813732 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811395    2548 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:31:07.813732 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811398    2548 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:31:07.813732 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811400    2548 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:31:07.813732 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811403    2548 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:31:07.813732 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811406    2548 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:31:07.813732 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811408    2548 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:31:07.813732 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811411    2548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:31:07.813732 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811414    2548 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:31:07.813732 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811416    2548 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:31:07.813732 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811419    2548 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:31:07.813732 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811421    2548 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:31:07.813732 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811423    2548 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:31:07.813732 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811426    2548 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:31:07.814217 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811429    2548 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:31:07.814217 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811431    2548 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:31:07.814217 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811434    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:31:07.814217 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811439    2548 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:31:07.814217 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811442    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:31:07.814217 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811444    2548 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:31:07.814217 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811447    2548 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:31:07.814217 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811449    2548 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:31:07.814217 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811452    2548 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:31:07.814217 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811454    2548 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:31:07.814217 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811456    2548 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:31:07.814217 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811459    2548 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:31:07.814217 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811461    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:31:07.814217 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811464    2548 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:31:07.814217 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811466    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:31:07.814217 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811988    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:31:07.814217 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811993    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:31:07.814217 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811996    2548 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:31:07.814217 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.811999    2548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:31:07.814692 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812002    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:31:07.814692 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812004    2548 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:31:07.814692 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812007    2548 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:31:07.814692 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812026    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:31:07.814692 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812031    2548 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:31:07.814692 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812035    2548 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:31:07.814692 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812037    2548 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:31:07.814692 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812040    2548 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:31:07.814692 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812043    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:31:07.814692 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812045    2548 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:31:07.814692 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812048    2548 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:31:07.814692 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812051    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:31:07.814692 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812054    2548 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:31:07.814692 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812056    2548 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:31:07.814692 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812059    2548 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:31:07.814692 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812062    2548 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:31:07.814692 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812065    2548 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:31:07.814692 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812068    2548 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:31:07.814692 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812070    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:31:07.814692 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812073    2548 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:31:07.815176 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812075    2548 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:31:07.815176 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812078    2548 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:31:07.815176 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812080    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:31:07.815176 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812082    2548 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:31:07.815176 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812085    2548 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:31:07.815176 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812087    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:31:07.815176 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812090    2548 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:31:07.815176 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812092    2548 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:31:07.815176 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812095    2548 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:31:07.815176 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812098    2548 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:31:07.815176 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812100    2548 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:31:07.815176 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812103    2548 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:31:07.815176 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812106    2548 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:31:07.815176 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812108    2548 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:31:07.815176 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812111    2548 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:31:07.815176 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812113    2548 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:31:07.815176 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812116    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:31:07.815176 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812118    2548 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:31:07.815176 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812121    2548 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:31:07.815176 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812124    2548 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:31:07.815665 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812126    2548 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:31:07.815665 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812129    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:31:07.815665 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812131    2548 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:31:07.815665 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812134    2548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:31:07.815665 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812136    2548 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:31:07.815665 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812139    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:31:07.815665 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812142    2548 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:31:07.815665 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812144    2548 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:31:07.815665 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812147    2548 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:31:07.815665 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812150    2548 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:31:07.815665 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812152    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:31:07.815665 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812155    2548 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:31:07.815665 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812157    2548 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:31:07.815665 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812160    2548 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:31:07.815665 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812162    2548 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:31:07.815665 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812165    2548 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:31:07.815665 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812167    2548 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:31:07.815665 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812169    2548 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:31:07.815665 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812172    2548 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:31:07.816113 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812175    2548 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:31:07.816113 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812178    2548 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:31:07.816113 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812180    2548 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:31:07.816113 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812183    2548 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:31:07.816113 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812186    2548 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:31:07.816113 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812188    2548 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:31:07.816113 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812191    2548 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:31:07.816113 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812193    2548 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:31:07.816113 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812197    2548 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:31:07.816113 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812201    2548 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:31:07.816113 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812204    2548 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:31:07.816113 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812208    2548 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:31:07.816113 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812211    2548 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:31:07.816113 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812214    2548 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:31:07.816113 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812217    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:31:07.816113 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812219    2548 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:31:07.816113 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812222    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:31:07.816113 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812224    2548 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:31:07.816113 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812227    2548 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:31:07.816576 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812229    2548 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:31:07.816576 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812232    2548 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:31:07.816576 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812234    2548 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:31:07.816576 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812236    2548 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:31:07.816576 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812239    2548 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:31:07.816576 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812241    2548 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:31:07.816576 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812244    2548 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:31:07.816576 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812246    2548 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:31:07.816576 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812249    2548 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:31:07.816576 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812251    2548 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:31:07.816576 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812253    2548 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:31:07.816576 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812256    2548 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:31:07.816576 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812258    2548 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:31:07.816576 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812261    2548 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:31:07.816576 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812264    2548 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:31:07.816576 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812266    2548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:31:07.816576 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812269    2548 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:31:07.816576 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812272    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:31:07.816576 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812274    2548 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:31:07.816576 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812277    2548 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:31:07.817049 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812279    2548 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:31:07.817049 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812282    2548 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:31:07.817049 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812284    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:31:07.817049 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.812287    2548 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:31:07.817049 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812429    2548 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 00:31:07.817049 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812443    2548 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 00:31:07.817049 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812451    2548 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 00:31:07.817049 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812455    2548 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 00:31:07.817049 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812460    2548 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 00:31:07.817049 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812463    2548 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 00:31:07.817049 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812468    2548 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 00:31:07.817049 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812473    2548 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 00:31:07.817049 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812476    2548 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 00:31:07.817049 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812479    2548 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 00:31:07.817049 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812484    2548 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 00:31:07.817049 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812487    2548 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 00:31:07.817049 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812490    2548 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 00:31:07.817049 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812493    2548 flags.go:64] FLAG: --cgroup-root=""
Oct 30 00:31:07.817049 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812496    2548 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 00:31:07.817049 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812499    2548 flags.go:64] FLAG: --client-ca-file=""
Oct 30 00:31:07.817049 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812502    2548 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 00:31:07.817049 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812505    2548 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 00:31:07.817049 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812509    2548 flags.go:64] FLAG: --cluster-domain=""
Oct 30 00:31:07.817602 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812515    2548 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 00:31:07.817602 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812518    2548 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 00:31:07.817602 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812522    2548 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 00:31:07.817602 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812525    2548 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 00:31:07.817602 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812530    2548 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 00:31:07.817602 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812533    2548 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 00:31:07.817602 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812536    2548 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 00:31:07.817602 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812540    2548 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 00:31:07.817602 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812543    2548 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 00:31:07.817602 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812546    2548 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 00:31:07.817602 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812549    2548 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 00:31:07.817602 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812552    2548 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 00:31:07.817602 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812556    2548 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 00:31:07.817602 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812560    2548 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 00:31:07.817602 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812563    2548 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 00:31:07.817602 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812566    2548 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 00:31:07.817602 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812569    2548 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 00:31:07.817602 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812572    2548 flags.go:64] FLAG: --enable-server="true"
Oct 30 00:31:07.817602 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812575    2548 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 00:31:07.817602 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812580    2548 flags.go:64] FLAG: --event-burst="100"
Oct 30 00:31:07.817602 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812583    2548 flags.go:64] FLAG: --event-qps="50"
Oct 30 00:31:07.817602 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812586    2548 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 00:31:07.817602 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812590    2548 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 00:31:07.817602 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812593    2548 flags.go:64] FLAG: --eviction-hard=""
Oct 30 00:31:07.818191 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812598    2548 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 00:31:07.818191 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812601    2548 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 00:31:07.818191 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812604    2548 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 00:31:07.818191 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812607    2548 flags.go:64] FLAG: --eviction-soft=""
Oct 30 00:31:07.818191 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812610    2548 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 00:31:07.818191 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812613    2548 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 00:31:07.818191 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812616    2548 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 00:31:07.818191 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812619    2548 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 00:31:07.818191 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812623    2548 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 00:31:07.818191 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812626    2548 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 00:31:07.818191 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812629    2548 flags.go:64] FLAG: --feature-gates=""
Oct 30 00:31:07.818191 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812634    2548 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 00:31:07.818191 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812637    2548 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 00:31:07.818191 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812640    2548 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 00:31:07.818191 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812643    2548 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 00:31:07.818191 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812647    2548 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 00:31:07.818191 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812650    2548 flags.go:64] FLAG: --help="false"
Oct 30 00:31:07.818191 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812653    2548 flags.go:64] FLAG: --hostname-override="ip-10-0-135-56.ec2.internal"
Oct 30 00:31:07.818191 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812656    2548 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 00:31:07.818191 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812659    2548 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 00:31:07.818191 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812662    2548 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 00:31:07.818191 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812665    2548 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 00:31:07.818191 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812668    2548 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 00:31:07.818759 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812671    2548 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 00:31:07.818759 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812674    2548 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 00:31:07.818759 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812677    2548 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 00:31:07.818759 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812679    2548 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 00:31:07.818759 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812682    2548 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 00:31:07.818759 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812685    2548 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 00:31:07.818759 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812688    2548 flags.go:64] FLAG: --kube-reserved=""
Oct 30 00:31:07.818759 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812691    2548 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 00:31:07.818759 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812694    2548 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 00:31:07.818759 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812697    2548 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 00:31:07.818759 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812699    2548 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 00:31:07.818759 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812702    2548 flags.go:64] FLAG: --lock-file=""
Oct 30 00:31:07.818759 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812705    2548 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 00:31:07.818759 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812708    2548 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 00:31:07.818759 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812711    2548 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 00:31:07.818759 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812716    2548 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 00:31:07.818759 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812719    2548 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 00:31:07.818759 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812724    2548 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 00:31:07.818759 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812727    2548 flags.go:64] FLAG: --logging-format="text"
Oct 30 00:31:07.818759 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812731    2548 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 00:31:07.818759 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812735    2548 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 00:31:07.818759 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812738    2548 flags.go:64] FLAG: --manifest-url=""
Oct 30 00:31:07.818759 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812741    2548 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 00:31:07.818759 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812746    2548 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 00:31:07.818759 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812749    2548 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 00:31:07.819367 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812753    2548 flags.go:64] FLAG: --max-pods="110"
Oct 30 00:31:07.819367 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812756    2548 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 00:31:07.819367 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812760    2548 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 00:31:07.819367 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812763    2548 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 00:31:07.819367 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812766    2548 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 00:31:07.819367 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812770    2548 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 00:31:07.819367 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812773    2548 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 00:31:07.819367 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812776    2548 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 00:31:07.819367 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812784    2548 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 00:31:07.819367 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812792    2548 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 00:31:07.819367 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812795    2548 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 00:31:07.819367 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812798    2548 flags.go:64] FLAG: --pod-cidr=""
Oct 30 00:31:07.819367 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812801    2548 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 00:31:07.819367 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812804    2548 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 00:31:07.819367 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812807    2548 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 00:31:07.819367 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812810    2548 flags.go:64] FLAG: --port="10250"
Oct 30 00:31:07.819367 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812813    2548 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 00:31:07.819367 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812816    2548 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-000c194909efd78be"
Oct 30 00:31:07.819367 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812819    2548 flags.go:64] FLAG: --qos-reserved=""
Oct 30 00:31:07.819367 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812822    2548 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 00:31:07.819367 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812825    2548 flags.go:64] FLAG: --register-node="true"
Oct 30 00:31:07.819367 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812828    2548 flags.go:64] FLAG: --register-with-taints=""
Oct 30 00:31:07.819367 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812832    2548 flags.go:64] FLAG: --registry-burst="10"
Oct 30 00:31:07.819367 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812835    2548 flags.go:64] FLAG: --registry-qps="5"
Oct 30 00:31:07.819367 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812840    2548 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 00:31:07.819367 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812843    2548 flags.go:64] FLAG: --reserved-memory=""
Oct 30 00:31:07.819983 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812849    2548 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 00:31:07.819983 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812852    2548 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 00:31:07.819983 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812855    2548 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 00:31:07.819983 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812858    2548 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 00:31:07.819983 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812861    2548 flags.go:64] FLAG: --runonce="false"
Oct 30 00:31:07.819983 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812864    2548 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 00:31:07.819983 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812867    2548 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 00:31:07.819983 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812870    2548 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 00:31:07.819983 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812874    2548 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 00:31:07.819983 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812877    2548 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 00:31:07.819983 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812880    2548 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 00:31:07.819983 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812883    2548 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 00:31:07.819983 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812886    2548 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 00:31:07.819983 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812889    2548 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 00:31:07.819983 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812892    2548 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 00:31:07.819983 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812895    2548 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 00:31:07.819983 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812898    2548 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 00:31:07.819983 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812901    2548 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 00:31:07.819983 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812904    2548 flags.go:64] FLAG: --system-cgroups=""
Oct 30 00:31:07.819983 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812906    2548 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 00:31:07.819983 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812912    2548 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 00:31:07.819983 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812915    2548 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 00:31:07.819983 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812918    2548 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 00:31:07.819983 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812922    2548 flags.go:64] FLAG: --tls-min-version=""
Oct 30 00:31:07.819983 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812925    2548 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 00:31:07.820588 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812928    2548 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 00:31:07.820588 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812931    2548 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 00:31:07.820588 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812934    2548 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 00:31:07.820588 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812937    2548 flags.go:64] FLAG: --v="2"
Oct 30 00:31:07.820588 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812942    2548 flags.go:64] FLAG: --version="false"
Oct 30 00:31:07.820588 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812947    2548 flags.go:64] FLAG: --vmodule=""
Oct 30 00:31:07.820588 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812952    2548 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 00:31:07.820588 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.812957    2548 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 00:31:07.820588 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813101    2548 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:31:07.820588 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813105    2548 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:31:07.820588 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813108    2548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:31:07.820588 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813111    2548 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:31:07.820588 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813113    2548 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:31:07.820588 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813116    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:31:07.820588 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813119    2548 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:31:07.820588 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813121    2548 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:31:07.820588 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813124    2548 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:31:07.820588 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813127    2548 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:31:07.820588 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813130    2548 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:31:07.820588 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813133    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:31:07.820588 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813135    2548 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:31:07.821158 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813138    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:31:07.821158 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813140    2548 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:31:07.821158 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813143    2548 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:31:07.821158 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813145    2548 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:31:07.821158 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813148    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:31:07.821158 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813150    2548 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:31:07.821158 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813153    2548 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:31:07.821158 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813155    2548 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:31:07.821158 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813157    2548 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:31:07.821158 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813160    2548 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:31:07.821158 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813162    2548 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:31:07.821158 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813165    2548 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:31:07.821158 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813167    2548 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:31:07.821158 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813170    2548 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:31:07.821158 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813172    2548 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:31:07.821158 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813175    2548 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:31:07.821158 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813179    2548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:31:07.821158 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813181    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:31:07.821158 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813185    2548 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:31:07.821158 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813188    2548 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:31:07.821660 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813190    2548 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:31:07.821660 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813193    2548 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:31:07.821660 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813195    2548 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:31:07.821660 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813198    2548 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:31:07.821660 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813200    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:31:07.821660 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813203    2548 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:31:07.821660 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813206    2548 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:31:07.821660 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813208    2548 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:31:07.821660 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813211    2548 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:31:07.821660 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813214    2548 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:31:07.821660 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813217    2548 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:31:07.821660 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813219    2548 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:31:07.821660 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813222    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:31:07.821660 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813224    2548 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:31:07.821660 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813227    2548 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:31:07.821660 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813229    2548 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:31:07.821660 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813232    2548 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:31:07.821660 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813234    2548 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:31:07.821660 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813237    2548 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:31:07.821660 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813239    2548 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:31:07.822117 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813242    2548 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:31:07.822117 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813245    2548 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:31:07.822117 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813248    2548 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:31:07.822117 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813251    2548 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:31:07.822117 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813253    2548 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:31:07.822117 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813256    2548 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:31:07.822117 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813259    2548 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:31:07.822117 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813261    2548 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:31:07.822117 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813265    2548 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:31:07.822117 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813267    2548 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:31:07.822117 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813271    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:31:07.822117 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813274    2548 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:31:07.822117 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813276    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:31:07.822117 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813279    2548 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:31:07.822117 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813281    2548 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:31:07.822117 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813284    2548 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:31:07.822117 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813286    2548 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:31:07.822117 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813289    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:31:07.822117 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813291    2548 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:31:07.822117 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813294    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:31:07.822652 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813296    2548 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:31:07.822652 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813299    2548 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:31:07.822652 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813302    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:31:07.822652 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813304    2548 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:31:07.822652 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813307    2548 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:31:07.822652 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813309    2548 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:31:07.822652 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813312    2548 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:31:07.822652 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813314    2548 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:31:07.822652 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813317    2548 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:31:07.822652 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813319    2548 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:31:07.822652 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813323    2548 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:31:07.822652 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813327    2548 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:31:07.822652 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813330    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:31:07.822652 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813333    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:31:07.822652 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813336    2548 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:31:07.822652 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813338    2548 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:31:07.822652 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813340    2548 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:31:07.822652 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813343    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:31:07.822652 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813345    2548 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:31:07.823127 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813348    2548 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:31:07.823127 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813365    2548 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:31:07.823127 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813368    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:31:07.823127 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813372    2548 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:31:07.823127 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813376    2548 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:31:07.823127 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813379    2548 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:31:07.823127 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813382    2548 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:31:07.823127 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813384    2548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:31:07.823127 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813387    2548 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:31:07.823127 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813390    2548 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:31:07.823127 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813393    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:31:07.823127 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813395    2548 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:31:07.823127 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813397    2548 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:31:07.823127 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813400    2548 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:31:07.823127 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.813428    2548 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:31:07.823127 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813472    2548 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:31:07.823551 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813477    2548 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:31:07.823551 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813480    2548 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:31:07.823551 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813484    2548 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:31:07.823551 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813486    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:31:07.823551 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813488    2548 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:31:07.823551 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813492    2548 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:31:07.823551 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813494    2548 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:31:07.823551 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813497    2548 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:31:07.823551 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813500    2548 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:31:07.823551 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813502    2548 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:31:07.823551 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813505    2548 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:31:07.823551 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813507    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:31:07.823551 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813510    2548 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:31:07.823551 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813512    2548 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:31:07.823551 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813515    2548 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:31:07.823551 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813517    2548 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:31:07.823551 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813520    2548 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:31:07.823551 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813525    2548 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:31:07.823551 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813527    2548 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:31:07.831585 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813530    2548 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:31:07.831585 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813534    2548 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:31:07.831585 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813537    2548 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:31:07.831585 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813540    2548 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:31:07.831585 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813542    2548 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:31:07.831585 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813545    2548 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:31:07.831585 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813547    2548 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:31:07.831585 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813550    2548 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:31:07.831585 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813552    2548 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:31:07.831585 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813555    2548 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:31:07.831585 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813557    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:31:07.831585 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813560    2548 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:31:07.831585 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813562    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:31:07.831585 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813566    2548 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:31:07.831585 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813568    2548 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:31:07.831585 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813571    2548 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:31:07.831585 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813573    2548 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:31:07.831585 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813583    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:31:07.831585 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813586    2548 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:31:07.831585 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813588    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:31:07.832421 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813591    2548 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:31:07.832421 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813593    2548 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:31:07.832421 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813596    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:31:07.832421 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813598    2548 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:31:07.832421 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813601    2548 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:31:07.832421 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813603    2548 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:31:07.832421 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813606    2548 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:31:07.832421 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813608    2548 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:31:07.832421 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813610    2548 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:31:07.832421 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813613    2548 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:31:07.832421 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813618    2548 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:31:07.832421 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813622    2548 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:31:07.832421 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813625    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:31:07.832421 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813629    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:31:07.832421 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813632    2548 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:31:07.832421 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813635    2548 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:31:07.832421 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813638    2548 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:31:07.832421 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813640    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:31:07.832421 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813643    2548 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:31:07.832961 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813645    2548 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:31:07.832961 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813648    2548 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:31:07.832961 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813650    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:31:07.832961 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813653    2548 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:31:07.832961 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813657    2548 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:31:07.832961 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813661    2548 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:31:07.832961 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813664    2548 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:31:07.832961 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813667    2548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:31:07.832961 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813670    2548 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:31:07.832961 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813674    2548 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:31:07.832961 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813677    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:31:07.832961 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813679    2548 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:31:07.832961 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813682    2548 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:31:07.832961 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813684    2548 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:31:07.832961 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813687    2548 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:31:07.832961 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813690    2548 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:31:07.832961 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813692    2548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:31:07.832961 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813695    2548 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:31:07.832961 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813697    2548 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:31:07.833509 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813700    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:31:07.833509 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813702    2548 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:31:07.833509 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813705    2548 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:31:07.833509 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813707    2548 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:31:07.833509 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813711    2548 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:31:07.833509 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813714    2548 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:31:07.833509 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813716    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:31:07.833509 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813721    2548 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:31:07.833509 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813723    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:31:07.833509 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813726    2548 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:31:07.833509 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813729    2548 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:31:07.833509 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813732    2548 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:31:07.833509 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813735    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:31:07.833509 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813737    2548 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:31:07.833509 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813740    2548 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:31:07.833509 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813742    2548 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:31:07.833509 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813745    2548 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:31:07.833509 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813748    2548 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:31:07.833509 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813750    2548 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:31:07.833509 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813753    2548 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:31:07.833977 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813756    2548 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:31:07.833977 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813759    2548 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:31:07.833977 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813762    2548 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:31:07.833977 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813764    2548 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:31:07.833977 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813767    2548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:31:07.833977 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813769    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:31:07.833977 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813772    2548 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:31:07.833977 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.813774    2548 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:31:07.833977 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.822958    2548 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 00:31:07.833977 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.822983    2548 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 00:31:07.833977 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.823596    2548 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 00:31:07.833977 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.823616    2548 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 00:31:07.833977 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823706    2548 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:31:07.833977 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823712    2548 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:31:07.833977 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823715    2548 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:31:07.833977 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823718    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:31:07.833977 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823721    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:31:07.833977 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823724    2548 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:31:07.833977 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823727    2548 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:31:07.834458 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823729    2548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:31:07.834458 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823732    2548 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:31:07.834458 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823735    2548 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:31:07.834458 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823737    2548 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:31:07.834458 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823740    2548 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:31:07.834458 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823743    2548 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:31:07.834458 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823745    2548 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:31:07.834458 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823748    2548 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:31:07.834458 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823750    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:31:07.834458 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823753    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:31:07.834458 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823755    2548 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:31:07.834458 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823757    2548 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:31:07.834458 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823760    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:31:07.834458 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823763    2548 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:31:07.834458 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823765    2548 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:31:07.834458 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823767    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:31:07.834458 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823770    2548 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:31:07.834458 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823772    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:31:07.834458 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823775    2548 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:31:07.834918 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823777    2548 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:31:07.834918 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823780    2548 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:31:07.834918 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823783    2548 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:31:07.834918 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823792    2548 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:31:07.834918 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823795    2548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:31:07.834918 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823798    2548 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:31:07.834918 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823801    2548 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:31:07.834918 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823803    2548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:31:07.834918 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823806    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:31:07.834918 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823808    2548 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:31:07.834918 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823811    2548 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:31:07.834918 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823814    2548 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:31:07.834918 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823816    2548 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:31:07.834918 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823819    2548 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:31:07.834918 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823821    2548 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:31:07.834918 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823824    2548 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:31:07.834918 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823827    2548 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:31:07.834918 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823829    2548 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:31:07.834918 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823834    2548 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:31:07.835512 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823838    2548 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:31:07.835512 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823841    2548 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:31:07.835512 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823844    2548 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:31:07.835512 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823847    2548 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:31:07.835512 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823850    2548 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:31:07.835512 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823853    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:31:07.835512 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823855    2548 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:31:07.835512 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823858    2548 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:31:07.835512 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823861    2548 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:31:07.835512 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823863    2548 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:31:07.835512 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823866    2548 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:31:07.835512 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823868    2548 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:31:07.835512 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823871    2548 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:31:07.835512 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823873    2548 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:31:07.835512 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823876    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:31:07.835512 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823879    2548 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:31:07.835512 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823881    2548 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:31:07.835512 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823890    2548 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:31:07.835512 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823893    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:31:07.836083 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823896    2548 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:31:07.836083 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823899    2548 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:31:07.836083 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823902    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:31:07.836083 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823905    2548 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:31:07.836083 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823907    2548 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:31:07.836083 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823910    2548 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:31:07.836083 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823913    2548 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:31:07.836083 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823915    2548 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:31:07.836083 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823919    2548 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:31:07.836083 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823923    2548 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:31:07.836083 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823926    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:31:07.836083 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823928    2548 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:31:07.836083 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823931    2548 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:31:07.836083 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823933    2548 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:31:07.836083 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823936    2548 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:31:07.836083 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823938    2548 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:31:07.836083 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823941    2548 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:31:07.836083 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823943    2548 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:31:07.836083 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823946    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:31:07.836624 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823948    2548 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:31:07.836624 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823951    2548 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:31:07.836624 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823953    2548 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:31:07.836624 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823956    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:31:07.836624 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823958    2548 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:31:07.836624 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823961    2548 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:31:07.836624 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823963    2548 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:31:07.836624 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823966    2548 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:31:07.836624 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823969    2548 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:31:07.836624 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823971    2548 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:31:07.836624 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823974    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:31:07.836624 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823977    2548 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:31:07.836624 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823985    2548 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:31:07.836624 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823988    2548 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:31:07.836624 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823990    2548 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:31:07.836624 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823993    2548 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:31:07.836624 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823995    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:31:07.836624 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.823998    2548 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:31:07.836624 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824000    2548 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:31:07.836624 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824003    2548 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:31:07.837116 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824005    2548 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:31:07.837116 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824008    2548 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:31:07.837116 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824010    2548 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:31:07.837116 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.824045    2548 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:31:07.837116 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824258    2548 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:31:07.837116 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824264    2548 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:31:07.837116 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824267    2548 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:31:07.837116 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824271    2548 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:31:07.837116 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824273    2548 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:31:07.837116 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824276    2548 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:31:07.837116 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824279    2548 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:31:07.837116 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824281    2548 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:31:07.837116 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824284    2548 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:31:07.837116 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824286    2548 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:31:07.837116 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824289    2548 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:31:07.837116 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824291    2548 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:31:07.837516 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824294    2548 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:31:07.837516 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824296    2548 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:31:07.837516 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824299    2548 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:31:07.837516 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824301    2548 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:31:07.837516 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824304    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:31:07.837516 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824307    2548 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:31:07.837516 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824310    2548 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:31:07.837516 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824312    2548 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:31:07.837516 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824315    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:31:07.837516 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824324    2548 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:31:07.837516 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824326    2548 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:31:07.837516 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824329    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:31:07.837516 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824331    2548 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:31:07.837516 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824334    2548 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:31:07.837516 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824336    2548 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:31:07.837516 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824339    2548 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:31:07.837516 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824341    2548 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:31:07.837516 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824344    2548 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:31:07.837516 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824346    2548 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:31:07.838039 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824349    2548 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:31:07.838039 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824370    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:31:07.838039 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824373    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:31:07.838039 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824376    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:31:07.838039 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824378    2548 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:31:07.838039 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824383    2548 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:31:07.838039 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824386    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:31:07.838039 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824389    2548 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:31:07.838039 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824392    2548 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:31:07.838039 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824395    2548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:31:07.838039 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824397    2548 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:31:07.838039 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824400    2548 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:31:07.838039 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824402    2548 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:31:07.838039 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824405    2548 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:31:07.838039 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824407    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:31:07.838039 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824410    2548 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:31:07.838039 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824412    2548 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:31:07.838039 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824415    2548 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:31:07.838039 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824418    2548 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:31:07.838524 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824420    2548 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:31:07.838524 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824423    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:31:07.838524 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824426    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:31:07.838524 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824429    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:31:07.838524 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824466    2548 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:31:07.838524 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824470    2548 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:31:07.838524 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824473    2548 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:31:07.838524 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824475    2548 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:31:07.838524 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824478    2548 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:31:07.838524 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824480    2548 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:31:07.838524 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824483    2548 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:31:07.838524 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824485    2548 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:31:07.838524 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824488    2548 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:31:07.838524 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824490    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:31:07.838524 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824493    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:31:07.838524 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824495    2548 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:31:07.838524 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824498    2548 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:31:07.838524 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824500    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:31:07.838524 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824503    2548 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:31:07.838524 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824506    2548 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:31:07.838994 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824508    2548 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:31:07.838994 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824511    2548 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:31:07.838994 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824513    2548 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:31:07.838994 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824516    2548 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:31:07.838994 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824519    2548 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:31:07.838994 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824527    2548 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:31:07.838994 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824530    2548 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:31:07.838994 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824532    2548 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:31:07.838994 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824535    2548 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:31:07.838994 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824537    2548 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:31:07.838994 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824540    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:31:07.838994 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824543    2548 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:31:07.838994 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824545    2548 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:31:07.838994 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824548    2548 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:31:07.838994 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824551    2548 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:31:07.838994 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824553    2548 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:31:07.838994 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824556    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:31:07.838994 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824564    2548 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:31:07.838994 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824567    2548 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:31:07.838994 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824569    2548 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:31:07.839498 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824572    2548 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:31:07.839498 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824574    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:31:07.839498 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824577    2548 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:31:07.839498 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824579    2548 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:31:07.839498 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824582    2548 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:31:07.839498 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824585    2548 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:31:07.839498 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824589    2548 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:31:07.839498 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824591    2548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:31:07.839498 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824594    2548 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:31:07.839498 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824597    2548 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:31:07.839498 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824599    2548 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:31:07.839498 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824602    2548 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:31:07.839498 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824604    2548 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:31:07.839498 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824607    2548 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:31:07.839498 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824609    2548 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:31:07.839498 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824612    2548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:31:07.839880 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.824638    2548 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:31:07.839880 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824676    2548 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:31:07.839880 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824680    2548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:31:07.839880 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824683    2548 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:31:07.839880 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824686    2548 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:31:07.839880 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824689    2548 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:31:07.839880 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824691    2548 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:31:07.839880 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824694    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:31:07.839880 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824697    2548 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:31:07.839880 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824699    2548 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:31:07.839880 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824702    2548 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:31:07.839880 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824704    2548 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:31:07.839880 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824707    2548 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:31:07.839880 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824709    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:31:07.839880 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824717    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:31:07.839880 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824720    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:31:07.840261 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824723    2548 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:31:07.840261 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824726    2548 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:31:07.840261 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824728    2548 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:31:07.840261 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824736    2548 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:31:07.840261 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824739    2548 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:31:07.840261 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824741    2548 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:31:07.840261 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824744    2548 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:31:07.840261 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824747    2548 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:31:07.840261 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824749    2548 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:31:07.840261 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824751    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:31:07.840261 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824754    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:31:07.840261 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824757    2548 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:31:07.840261 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824759    2548 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:31:07.840261 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824761    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:31:07.840261 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824764    2548 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:31:07.840261 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824766    2548 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:31:07.840261 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824769    2548 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:31:07.840261 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824772    2548 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:31:07.840261 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824774    2548 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:31:07.840261 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824777    2548 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:31:07.840747 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824779    2548 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:31:07.840747 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824782    2548 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:31:07.840747 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824784    2548 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:31:07.840747 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824786    2548 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:31:07.840747 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824790    2548 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:31:07.840747 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824792    2548 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:31:07.840747 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824795    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:31:07.840747 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824797    2548 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:31:07.840747 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824800    2548 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:31:07.840747 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824802    2548 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:31:07.840747 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824805    2548 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:31:07.840747 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824813    2548 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:31:07.840747 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824816    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:31:07.840747 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824818    2548 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:31:07.840747 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824821    2548 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:31:07.840747 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824825    2548 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:31:07.840747 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824827    2548 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:31:07.840747 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824830    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:31:07.840747 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824832    2548 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:31:07.840747 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824835    2548 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:31:07.841209 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824837    2548 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:31:07.841209 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824841    2548 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:31:07.841209 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824845    2548 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:31:07.841209 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824848    2548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:31:07.841209 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824851    2548 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:31:07.841209 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824853    2548 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:31:07.841209 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824856    2548 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:31:07.841209 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824858    2548 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:31:07.841209 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824861    2548 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:31:07.841209 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824863    2548 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:31:07.841209 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824865    2548 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:31:07.841209 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824868    2548 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:31:07.841209 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824870    2548 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:31:07.841209 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824873    2548 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:31:07.841209 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824875    2548 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:31:07.841209 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824879    2548 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:31:07.841209 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824882    2548 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:31:07.841209 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824884    2548 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:31:07.841209 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824887    2548 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:31:07.841669 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824889    2548 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:31:07.841669 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824892    2548 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:31:07.841669 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824895    2548 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:31:07.841669 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824897    2548 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:31:07.841669 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824900    2548 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:31:07.841669 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824917    2548 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:31:07.841669 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824920    2548 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:31:07.841669 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824923    2548 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:31:07.841669 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824927    2548 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:31:07.841669 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824929    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:31:07.841669 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824932    2548 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:31:07.841669 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824934    2548 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:31:07.841669 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824937    2548 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:31:07.841669 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824939    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:31:07.841669 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824942    2548 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:31:07.841669 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824944    2548 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:31:07.841669 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824947    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:31:07.841669 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824949    2548 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:31:07.841669 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824952    2548 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:31:07.842095 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824954    2548 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:31:07.842095 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824957    2548 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:31:07.842095 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824959    2548 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:31:07.842095 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824962    2548 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:31:07.842095 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824964    2548 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:31:07.842095 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824967    2548 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:31:07.842095 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824972    2548 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:31:07.842095 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824974    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:31:07.842095 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824977    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:31:07.842095 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824980    2548 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:31:07.842095 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824984    2548 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:31:07.842095 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824988    2548 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:31:07.842095 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:07.824992    2548 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:31:07.842095 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.825055    2548 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 00:31:07.842095 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.827094    2548 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 00:31:07.842095 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.827948    2548 server.go:1014] "Starting client certificate rotation"
Oct 30 00:31:07.842095 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.829325    2548 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 00:31:07.842095 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.829454    2548 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 00:31:07.842545 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.832680    2548 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 00:31:07.842545 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.834449    2548 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 00:31:07.843706 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.843686    2548 log.go:25] "Validated CRI v1 runtime API"
Oct 30 00:31:07.846697 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.846679    2548 log.go:25] "Validated CRI v1 image API"
Oct 30 00:31:07.847674 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.847647    2548 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 00:31:07.850021 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.849997    2548 fs.go:135] Filesystem UUIDs: map[3de2431c-c72f-4541-adb5-2d0f804e005b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ff55ea1e-4b8a-433f-b65f-72d114147a4e:/dev/nvme0n1p3]
Oct 30 00:31:07.850115 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.850019    2548 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 00:31:07.855792 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.855655    2548 manager.go:217] Machine: {Timestamp:2026-10-30 00:31:07.853841064 +0000 UTC m=+0.300080296 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3133592 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c96115e1e07bbe1f2949e21fc3a18 SystemUUID:ec2c9611-5e1e-07bb-e1f2-949e21fc3a18 BootID:faa9a921-8373-404e-8241-d554d5495c82 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:dd:6f:ac:a5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:dd:6f:ac:a5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:c1:57:0f:ad:1b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 00:31:07.855792 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.855780    2548 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 00:31:07.855987 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.855969    2548 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 00:31:07.858724 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.858692    2548 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 00:31:07.858872 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.858727    2548 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-56.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 00:31:07.858915 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.858875    2548 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 00:31:07.858915 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.858884    2548 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 00:31:07.858964 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.858932    2548 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 00:31:07.858964 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.858950    2548 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 00:31:07.859677 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.859666    2548 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 00:31:07.862694 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.862682    2548 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 00:31:07.862825 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.862815    2548 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 00:31:07.863452 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.863442    2548 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 00:31:07.863490 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.863458    2548 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 00:31:07.863490 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.863469    2548 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 00:31:07.863490 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.863480    2548 kubelet.go:405] "Adding apiserver pod source"
Oct 30 00:31:07.863490 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.863491    2548 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 00:31:07.863915 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.863903    2548 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 00:31:07.865942 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.865921    2548 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 00:31:07.872415 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.871568    2548 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 00:31:07.872997 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.872766    2548 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 00:31:07.873051 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.873031    2548 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 00:31:07.873277 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.873266    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 00:31:07.873309 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.873284    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 00:31:07.873309 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.873294    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 00:31:07.873309 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.873302    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 00:31:07.873408 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.873310    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 00:31:07.873408 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.873317    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 00:31:07.873823 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.873814    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 00:31:07.873852 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.873824    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 00:31:07.873852 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.873833    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 00:31:07.873852 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.873839    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 00:31:07.873852 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.873848    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 00:31:07.874049 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.873858    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 00:31:07.874824 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.874814    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 00:31:07.874862 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.874826    2548 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 00:31:07.878747 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.878727    2548 server.go:1263] "Started kubelet"
Oct 30 00:31:07.878884 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.878815    2548 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 00:31:07.878884 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.878836    2548 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 00:31:07.879016 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.878921    2548 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 00:31:07.879765 ip-10-0-135-56 systemd[1]: Started Kubernetes Kubelet.
Oct 30 00:31:07.880461 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.879943    2548 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 00:31:07.881223 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.881212    2548 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 00:31:07.883966 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:07.883945    2548 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 00:31:07.886373 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.886337    2548 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 00:31:07.886373 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.886347    2548 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 00:31:07.886500 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:07.886444    2548 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-56.ec2.internal\" not found"
Oct 30 00:31:07.886870 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.886850    2548 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 00:31:07.886870 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.886853    2548 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 00:31:07.887013 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.886876    2548 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 00:31:07.887013 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.886997    2548 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 00:31:07.887013 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.887007    2548 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 00:31:07.887255 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.887229    2548 factory.go:55] Registering systemd factory
Oct 30 00:31:07.887255 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.887248    2548 factory.go:223] Registration of the systemd container factory successfully
Oct 30 00:31:07.887591 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.887573    2548 factory.go:153] Registering CRI-O factory
Oct 30 00:31:07.887591 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.887588    2548 factory.go:223] Registration of the crio container factory successfully
Oct 30 00:31:07.887747 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.887645    2548 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 00:31:07.887747 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.887667    2548 factory.go:103] Registering Raw factory
Oct 30 00:31:07.887747 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.887683    2548 manager.go:1199] Started watching for new ooms in manager
Oct 30 00:31:07.888557 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.888540    2548 manager.go:319] Starting recovery of all containers
Oct 30 00:31:07.899732 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.899707    2548 manager.go:324] Recovery completed
Oct 30 00:31:07.907379 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.907343    2548 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:31:07.909188 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.909160    2548 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-56.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:31:07.909288 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.909198    2548 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-56.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:31:07.909288 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.909209    2548 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-56.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:31:07.909832 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.909817    2548 cpu_manager.go:226] "Starting" policy="none"
Oct 30 00:31:07.909875 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.909834    2548 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 00:31:07.909875 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.909856    2548 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 00:31:07.909934 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.909822    2548 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:31:07.910673 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.910650    2548 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-56.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:31:07.910761 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.910688    2548 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-56.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:31:07.910761 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.910702    2548 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-56.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:31:07.911485 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.911472    2548 policy_none.go:50] "Start"
Oct 30 00:31:07.911530 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.911488    2548 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 00:31:07.911530 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.911499    2548 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 00:31:07.912521 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.912511    2548 policy_none.go:44] "Start"
Oct 30 00:31:07.913634 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:07.913608    2548 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-56.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 00:31:07.913634 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.913616    2548 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-56.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 00:31:07.914460 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:07.913600    2548 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-56.ec2.internal.18e327483f6b0471  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-56.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-56.ec2.internal,},FirstTimestamp:2026-10-30 00:31:07.878696049 +0000 UTC m=+0.324935279,LastTimestamp:2026-10-30 00:31:07.878696049 +0000 UTC m=+0.324935279,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-56.ec2.internal,}"
Oct 30 00:31:07.919402 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.919381    2548 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 00:31:07.921379 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.921343    2548 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-txc6r"
Oct 30 00:31:07.933268 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.933028    2548 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-txc6r"
Oct 30 00:31:07.933268 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:07.933244    2548 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 00:31:07.933268 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:07.933122    2548 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-56.ec2.internal.18e32748413c3caa  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-56.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-56.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-56.ec2.internal,},FirstTimestamp:2026-10-30 00:31:07.909184682 +0000 UTC m=+0.355423911,LastTimestamp:2026-10-30 00:31:07.909184682 +0000 UTC m=+0.355423911,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-56.ec2.internal,}"
Oct 30 00:31:07.933555 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:07.933293    2548 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-56.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 00:31:07.933555 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:07.933247    2548 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 00:31:07.961156 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.957309    2548 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 00:31:07.961156 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.958051    2548 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 00:31:07.961156 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.959478    2548 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 00:31:07.961156 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.959501    2548 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 00:31:07.961156 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.959536    2548 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 00:31:07.961156 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:07.959570    2548 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 00:31:07.967971 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.967952    2548 manager.go:341] "Starting Device Plugin manager"
Oct 30 00:31:07.968108 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:07.967984    2548 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 00:31:07.968108 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.967993    2548 server.go:87] "Starting device plugin registration server"
Oct 30 00:31:07.968279 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.968267    2548 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 00:31:07.968318 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.968283    2548 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 00:31:07.968403 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.968386    2548 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 00:31:07.968518 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.968501    2548 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 00:31:07.968518 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.968513    2548 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 00:31:07.969011 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:07.968992    2548 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 00:31:07.969089 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:07.969037    2548 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-56.ec2.internal\" not found"
Oct 30 00:31:07.973061 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:07.973037    2548 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:31:08.059775 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.059681    2548 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-56.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal"]
Oct 30 00:31:08.059775 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.059752    2548 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:31:08.060904 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.060883    2548 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-56.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:31:08.061041 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.060927    2548 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-56.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:31:08.061041 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.060942    2548 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-56.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:31:08.062169 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.062151    2548 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:31:08.062315 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.062301    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-56.ec2.internal"
Oct 30 00:31:08.062418 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.062331    2548 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:31:08.062908 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.062892    2548 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-56.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:31:08.062979 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.062912    2548 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-56.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:31:08.062979 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.062921    2548 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-56.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:31:08.062979 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.062929    2548 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-56.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:31:08.062979 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.062932    2548 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-56.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:31:08.063092 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.062942    2548 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-56.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:31:08.064056 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.064043    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal"
Oct 30 00:31:08.064102 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.064078    2548 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:31:08.064825 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.064807    2548 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-56.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:31:08.064897 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.064838    2548 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-56.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:31:08.064897 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.064851    2548 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-56.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:31:08.068384 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.068371    2548 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:31:08.069132 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.069114    2548 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-56.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:31:08.069229 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.069149    2548 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-56.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:31:08.069229 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.069163    2548 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-56.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:31:08.069229 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.069195    2548 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-56.ec2.internal"
Oct 30 00:31:08.085397 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.085369    2548 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-56.ec2.internal"
Oct 30 00:31:08.085544 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:08.085405    2548 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-56.ec2.internal\": node \"ip-10-0-135-56.ec2.internal\" not found"
Oct 30 00:31:08.088440 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.088421    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/21539217511a9b702eb77d59ab0b339d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-56.ec2.internal\" (UID: \"21539217511a9b702eb77d59ab0b339d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-56.ec2.internal"
Oct 30 00:31:08.088486 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.088448    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f3ebcfeede45130240ea614a61281a18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal\" (UID: \"f3ebcfeede45130240ea614a61281a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal"
Oct 30 00:31:08.088486 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.088464    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3ebcfeede45130240ea614a61281a18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal\" (UID: \"f3ebcfeede45130240ea614a61281a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal"
Oct 30 00:31:08.095265 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:08.095243    2548 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-56.ec2.internal\" not found" node="ip-10-0-135-56.ec2.internal"
Oct 30 00:31:08.100657 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:08.100639    2548 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-56.ec2.internal\" not found" node="ip-10-0-135-56.ec2.internal"
Oct 30 00:31:08.118049 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:08.118026    2548 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-56.ec2.internal\" not found"
Oct 30 00:31:08.188743 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.188713    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/21539217511a9b702eb77d59ab0b339d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-56.ec2.internal\" (UID: \"21539217511a9b702eb77d59ab0b339d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-56.ec2.internal"
Oct 30 00:31:08.188743 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.188745    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f3ebcfeede45130240ea614a61281a18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal\" (UID: \"f3ebcfeede45130240ea614a61281a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal"
Oct 30 00:31:08.188743 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.188762    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3ebcfeede45130240ea614a61281a18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal\" (UID: \"f3ebcfeede45130240ea614a61281a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal"
Oct 30 00:31:08.188993 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.188810    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/21539217511a9b702eb77d59ab0b339d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-56.ec2.internal\" (UID: \"21539217511a9b702eb77d59ab0b339d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-56.ec2.internal"
Oct 30 00:31:08.188993 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.188829    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3ebcfeede45130240ea614a61281a18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal\" (UID: \"f3ebcfeede45130240ea614a61281a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal"
Oct 30 00:31:08.188993 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.188840    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f3ebcfeede45130240ea614a61281a18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal\" (UID: \"f3ebcfeede45130240ea614a61281a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal"
Oct 30 00:31:08.218629 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:08.218595    2548 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-56.ec2.internal\" not found"
Oct 30 00:31:08.319196 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:08.319108    2548 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-56.ec2.internal\" not found"
Oct 30 00:31:08.398630 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.398595    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-56.ec2.internal"
Oct 30 00:31:08.404101 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.404082    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal"
Oct 30 00:31:08.419281 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:08.419253    2548 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-56.ec2.internal\" not found"
Oct 30 00:31:08.519870 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:08.519827    2548 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-56.ec2.internal\" not found"
Oct 30 00:31:08.620438 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:08.620336    2548 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-56.ec2.internal\" not found"
Oct 30 00:31:08.721013 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:08.720982    2548 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-56.ec2.internal\" not found"
Oct 30 00:31:08.821545 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:08.821508    2548 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-56.ec2.internal\" not found"
Oct 30 00:31:08.827690 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.827647    2548 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 00:31:08.827844 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.827806    2548 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 00:31:08.905143 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.905079    2548 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:31:08.936512 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.936474    2548 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 00:26:09 +0000 UTC" deadline="2028-06-06 14:09:46.947627608 +0000 UTC"
Oct 30 00:31:08.936512 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.936511    2548 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14053h38m38.011121768s"
Oct 30 00:31:08.944231 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:08.944188    2548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3ebcfeede45130240ea614a61281a18.slice/crio-fd5af83b2e76ace40a2039e61ee3ae28927cf3ab03050ba4dedbb8c116e48408 WatchSource:0}: Error finding container fd5af83b2e76ace40a2039e61ee3ae28927cf3ab03050ba4dedbb8c116e48408: Status 404 returned error can't find the container with id fd5af83b2e76ace40a2039e61ee3ae28927cf3ab03050ba4dedbb8c116e48408
Oct 30 00:31:08.944745 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:08.944727    2548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21539217511a9b702eb77d59ab0b339d.slice/crio-62d17e940e9de9de17861e94f0b8198b5892d3baa6cda7fe581b304e78da0ef6 WatchSource:0}: Error finding container 62d17e940e9de9de17861e94f0b8198b5892d3baa6cda7fe581b304e78da0ef6: Status 404 returned error can't find the container with id 62d17e940e9de9de17861e94f0b8198b5892d3baa6cda7fe581b304e78da0ef6
Oct 30 00:31:08.951031 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.951013    2548 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:31:08.962889 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.962832    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-56.ec2.internal" event={"ID":"21539217511a9b702eb77d59ab0b339d","Type":"ContainerStarted","Data":"62d17e940e9de9de17861e94f0b8198b5892d3baa6cda7fe581b304e78da0ef6"}
Oct 30 00:31:08.963948 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.963919    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal" event={"ID":"f3ebcfeede45130240ea614a61281a18","Type":"ContainerStarted","Data":"fd5af83b2e76ace40a2039e61ee3ae28927cf3ab03050ba4dedbb8c116e48408"}
Oct 30 00:31:08.980493 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.980461    2548 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:31:08.987493 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:08.987464    2548 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-56.ec2.internal"
Oct 30 00:31:09.006364 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:09.006335    2548 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:31:09.007205 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:09.007191    2548 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal"
Oct 30 00:31:09.026110 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:09.026086    2548 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:31:09.291575 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:09.291492    2548 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:31:11.463423 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.463344    2548 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:31:11.684006 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.683969    2548 apiserver.go:52] "Watching apiserver"
Oct 30 00:31:11.706244 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.706213    2548 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 00:31:11.716709 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.716636    2548 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 00:31:11.717118 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.717095    2548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-56.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal","openshift-multus/multus-r58sp","openshift-cluster-node-tuning-operator/tuned-lnnlc","openshift-multus/network-metrics-daemon-jldmt","openshift-network-operator/iptables-alerter-s72l8","openshift-ovn-kubernetes/ovnkube-node-pxfhp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz","openshift-dns/node-resolver-fqtzs","openshift-image-registry/node-ca-9hslh","openshift-multus/multus-additional-cni-plugins-csc84","kube-system/global-pull-secret-syncer-f27vk","openshift-network-diagnostics/network-check-target-wpvjg","kube-system/konnectivity-agent-6wzd9"]
Oct 30 00:31:11.720492 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.720470    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.722666 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.722643    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9hslh"
Oct 30 00:31:11.724691 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.724669    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.727377 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.727360    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.731524 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.731023    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6wzd9"
Oct 30 00:31:11.733159 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.733137    2548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 00:31:11.733159 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.733152    2548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 00:31:11.733317 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.733207    2548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 00:31:11.733317 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.733218    2548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 00:31:11.733415 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.733349    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s72l8"
Oct 30 00:31:11.733456 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.733425    2548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-h6pr4\""
Oct 30 00:31:11.733508 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.733463    2548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 00:31:11.733831 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.733814    2548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 00:31:11.735645 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.735624    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fqtzs"
Oct 30 00:31:11.737391 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.737371    2548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6rb57\""
Oct 30 00:31:11.737750 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.737720    2548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 00:31:11.740350 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.740328    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpvjg"
Oct 30 00:31:11.740460 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.740444    2548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 00:31:11.740666 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:11.740640    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpvjg" podUID="b3705749-d093-4b49-878d-e4616e81a450"
Oct 30 00:31:11.740718 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.740710    2548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 00:31:11.741056 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.741040    2548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8g889\""
Oct 30 00:31:11.741144 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.741126    2548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 00:31:11.741194 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.741180    2548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 00:31:11.742879 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.742854    2548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 00:31:11.743310 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.742922    2548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-ttql7\""
Oct 30 00:31:11.743310 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.742965    2548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:31:11.743310 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.743183    2548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 00:31:11.743310 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.743264    2548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 00:31:11.743652 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.743278    2548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 00:31:11.743761 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.743714    2548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-z22kq\""
Oct 30 00:31:11.743835 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.743821    2548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 00:31:11.744584 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.744159    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:11.744989 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.744969    2548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 00:31:11.745196 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.745178    2548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 00:31:11.745275 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.745261    2548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8njw4\""
Oct 30 00:31:11.745356 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.745342    2548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:31:11.746893 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.746876    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jldmt"
Oct 30 00:31:11.747015 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:11.746970    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jldmt" podUID="e552ba30-5b09-48a7-bffb-6a65a094c2bc"
Oct 30 00:31:11.749003 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.748978    2548 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 00:31:11.749106 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.749058    2548 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jxstz"
Oct 30 00:31:11.749312 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.749298    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:11.751921 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.751810    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:11.751921 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:11.751886    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f27vk" podUID="3b0df054-0749-4417-bb8b-b12b3353d25e"
Oct 30 00:31:11.759537 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.759514    2548 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jxstz"
Oct 30 00:31:11.808151 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.808124    2548 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 00:31:11.819533 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.819501    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-etc-kubernetes\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.819533 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.819542    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-ovn-node-metrics-cert\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.819743 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.819572    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-hostroot\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.819743 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.819623    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-var-lib-kubelet\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.819743 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.819672    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-run-openvswitch\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.819891 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.819748    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-run-ovn\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.819891 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.819811    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b734765-8e7f-4d8e-a74b-bd179adf33ee-cni-binary-copy\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.819891 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.819837    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-host-var-lib-kubelet\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.819891 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.819883    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-etc-sysctl-conf\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.820082 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.819919    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4a99683d-3f83-465d-9152-2f422641e087-iptables-alerter-script\") pod \"iptables-alerter-s72l8\" (UID: \"4a99683d-3f83-465d-9152-2f422641e087\") " pod="openshift-network-operator/iptables-alerter-s72l8"
Oct 30 00:31:11.820082 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.819968    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/05a614f5-3112-4216-b894-68bf3945458c-hosts-file\") pod \"node-resolver-fqtzs\" (UID: \"05a614f5-3112-4216-b894-68bf3945458c\") " pod="openshift-dns/node-resolver-fqtzs"
Oct 30 00:31:11.820082 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.820012    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ef9a0a39-051f-45a1-acc9-354a23e5b00b-system-cni-dir\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:11.820082 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.820050    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05a614f5-3112-4216-b894-68bf3945458c-tmp-dir\") pod \"node-resolver-fqtzs\" (UID: \"05a614f5-3112-4216-b894-68bf3945458c\") " pod="openshift-dns/node-resolver-fqtzs"
Oct 30 00:31:11.820245 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.820097    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-run-netns\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.820245 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.820137    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-cni-bin\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.820245 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.820169    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.820245 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.820207    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-system-cni-dir\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.820245 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.820238    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xvts\" (UniqueName: \"kubernetes.io/projected/b3705749-d093-4b49-878d-e4616e81a450-kube-api-access-7xvts\") pod \"network-check-target-wpvjg\" (UID: \"b3705749-d093-4b49-878d-e4616e81a450\") " pod="openshift-network-diagnostics/network-check-target-wpvjg"
Oct 30 00:31:11.820411 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.820263    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-sys-fs\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:11.820411 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.820300    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30e43c48-92a7-4c68-887c-9aaf7d6d1e1b-host\") pod \"node-ca-9hslh\" (UID: \"30e43c48-92a7-4c68-887c-9aaf7d6d1e1b\") " pod="openshift-image-registry/node-ca-9hslh"
Oct 30 00:31:11.820411 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.820325    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ef9a0a39-051f-45a1-acc9-354a23e5b00b-cni-binary-copy\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:11.820411 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.820350    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ef9a0a39-051f-45a1-acc9-354a23e5b00b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:11.820411 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.820381    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pchtx\" (UniqueName: \"kubernetes.io/projected/05a614f5-3112-4216-b894-68bf3945458c-kube-api-access-pchtx\") pod \"node-resolver-fqtzs\" (UID: \"05a614f5-3112-4216-b894-68bf3945458c\") " pod="openshift-dns/node-resolver-fqtzs"
Oct 30 00:31:11.820411 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.820406    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e552ba30-5b09-48a7-bffb-6a65a094c2bc-metrics-certs\") pod \"network-metrics-daemon-jldmt\" (UID: \"e552ba30-5b09-48a7-bffb-6a65a094c2bc\") " pod="openshift-multus/network-metrics-daemon-jldmt"
Oct 30 00:31:11.820596 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.820431    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-cni-netd\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.820596 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.820467    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/30e43c48-92a7-4c68-887c-9aaf7d6d1e1b-serviceca\") pod \"node-ca-9hslh\" (UID: \"30e43c48-92a7-4c68-887c-9aaf7d6d1e1b\") " pod="openshift-image-registry/node-ca-9hslh"
Oct 30 00:31:11.820596 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.820490    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-host\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.820596 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.820513    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-device-dir\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:11.820596 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.820574    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpfcq\" (UniqueName: \"kubernetes.io/projected/e552ba30-5b09-48a7-bffb-6a65a094c2bc-kube-api-access-gpfcq\") pod \"network-metrics-daemon-jldmt\" (UID: \"e552ba30-5b09-48a7-bffb-6a65a094c2bc\") " pod="openshift-multus/network-metrics-daemon-jldmt"
Oct 30 00:31:11.820767 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.820601    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-systemd-units\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.820767 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.820629    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-slash\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.820767 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.820649    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-host-var-lib-cni-bin\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.820767 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.820664    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-multus-conf-dir\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.820767 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.820697    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-etc-modprobe-d\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.820767 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.820719    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-etc-systemd\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.820767 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.820738    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a99683d-3f83-465d-9152-2f422641e087-host-slash\") pod \"iptables-alerter-s72l8\" (UID: \"4a99683d-3f83-465d-9152-2f422641e087\") " pod="openshift-network-operator/iptables-alerter-s72l8"
Oct 30 00:31:11.820767 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.820752    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:11.821096 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.820793    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-etc-openvswitch\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.821096 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.820866    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-ovnkube-config\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.821096 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.820897    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-multus-socket-dir-parent\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.821096 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.820941    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-etc-sysconfig\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.821096 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.820979    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-sys\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.821096 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.821020    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bgt8\" (UniqueName: \"kubernetes.io/projected/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-kube-api-access-4bgt8\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:11.821096 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.821045    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-run-ovn-kubernetes\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.821096 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.821063    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-host-run-k8s-cni-cncf-io\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.821405 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.821153    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-host-run-netns\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.821405 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.821198    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5687019d-945a-4307-8269-5c0ae6f377ec-tmp\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.821405 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.821229    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-socket-dir\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:11.821405 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.821263    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-registration-dir\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:11.821405 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.821288    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ef9a0a39-051f-45a1-acc9-354a23e5b00b-cnibin\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:11.821405 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.821305    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3b0df054-0749-4417-bb8b-b12b3353d25e-kubelet-config\") pod \"global-pull-secret-syncer-f27vk\" (UID: \"3b0df054-0749-4417-bb8b-b12b3353d25e\") " pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:11.821405 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.821321    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b0df054-0749-4417-bb8b-b12b3353d25e-original-pull-secret\") pod \"global-pull-secret-syncer-f27vk\" (UID: \"3b0df054-0749-4417-bb8b-b12b3353d25e\") " pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:11.821405 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.821353    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-etc-sysctl-d\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.821405 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.821373    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-node-log\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.821405 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.821389    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-env-overrides\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.821405 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.821405    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-ovnkube-script-lib\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.821763 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.821419    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3b0df054-0749-4417-bb8b-b12b3353d25e-dbus\") pod \"global-pull-secret-syncer-f27vk\" (UID: \"3b0df054-0749-4417-bb8b-b12b3353d25e\") " pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:11.821763 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.821442    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-cnibin\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.821763 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.821467    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-host-var-lib-cni-multus\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.821763 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.821492    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-host-run-multus-certs\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.821763 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.821539    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5687019d-945a-4307-8269-5c0ae6f377ec-etc-tuned\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.821763 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.821565    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4f7ca654-edeb-4457-9239-8da741b5dafd-konnectivity-ca\") pod \"konnectivity-agent-6wzd9\" (UID: \"4f7ca654-edeb-4457-9239-8da741b5dafd\") " pod="kube-system/konnectivity-agent-6wzd9"
Oct 30 00:31:11.821763 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.821587    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28lgt\" (UniqueName: \"kubernetes.io/projected/4a99683d-3f83-465d-9152-2f422641e087-kube-api-access-28lgt\") pod \"iptables-alerter-s72l8\" (UID: \"4a99683d-3f83-465d-9152-2f422641e087\") " pod="openshift-network-operator/iptables-alerter-s72l8"
Oct 30 00:31:11.821763 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.821605    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-etc-selinux\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:11.821763 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.821624    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-kubelet\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.821763 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.821643    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-log-socket\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.821763 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.821663    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wblgx\" (UniqueName: \"kubernetes.io/projected/ef9a0a39-051f-45a1-acc9-354a23e5b00b-kube-api-access-wblgx\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:11.821763 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.821747    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3b734765-8e7f-4d8e-a74b-bd179adf33ee-multus-daemon-config\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.822292 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.821792    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4f7ca654-edeb-4457-9239-8da741b5dafd-agent-certs\") pod \"konnectivity-agent-6wzd9\" (UID: \"4f7ca654-edeb-4457-9239-8da741b5dafd\") " pod="kube-system/konnectivity-agent-6wzd9"
Oct 30 00:31:11.822292 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.821851    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-run-systemd\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.822292 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.821875    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-etc-kubernetes\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.822292 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.821897    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-lib-modules\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.822292 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.821923    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqzmf\" (UniqueName: \"kubernetes.io/projected/5687019d-945a-4307-8269-5c0ae6f377ec-kube-api-access-gqzmf\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.822292 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.821980    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-var-lib-openvswitch\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.822292 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.822010    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgw94\" (UniqueName: \"kubernetes.io/projected/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-kube-api-access-bgw94\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.822292 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.822033    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-run\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.822292 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.822058    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ef9a0a39-051f-45a1-acc9-354a23e5b00b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:11.822292 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.822083    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-multus-cni-dir\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.822292 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.822146    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzcz2\" (UniqueName: \"kubernetes.io/projected/30e43c48-92a7-4c68-887c-9aaf7d6d1e1b-kube-api-access-zzcz2\") pod \"node-ca-9hslh\" (UID: \"30e43c48-92a7-4c68-887c-9aaf7d6d1e1b\") " pod="openshift-image-registry/node-ca-9hslh"
Oct 30 00:31:11.822292 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.822178    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ef9a0a39-051f-45a1-acc9-354a23e5b00b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:11.822292 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.822205    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7rhg\" (UniqueName: \"kubernetes.io/projected/3b734765-8e7f-4d8e-a74b-bd179adf33ee-kube-api-access-k7rhg\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.833837 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.833804    2548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mt5kv\""
Oct 30 00:31:11.833837 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.833804    2548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 00:31:11.834107 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.833956    2548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 00:31:11.834107 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.833974    2548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 00:31:11.834107 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.834022    2548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 00:31:11.834107 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.834089    2548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-d7h4r\""
Oct 30 00:31:11.834309 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.834117    2548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 00:31:11.834309 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.834218    2548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xlz4l\""
Oct 30 00:31:11.834309 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.834293    2548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 00:31:11.834572 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.834551    2548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 00:31:11.922694 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.922653    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-etc-sysctl-conf\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.922891 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.922708    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4a99683d-3f83-465d-9152-2f422641e087-iptables-alerter-script\") pod \"iptables-alerter-s72l8\" (UID: \"4a99683d-3f83-465d-9152-2f422641e087\") " pod="openshift-network-operator/iptables-alerter-s72l8"
Oct 30 00:31:11.922891 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.922879    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-etc-sysctl-conf\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.922972 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.922925    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/05a614f5-3112-4216-b894-68bf3945458c-hosts-file\") pod \"node-resolver-fqtzs\" (UID: \"05a614f5-3112-4216-b894-68bf3945458c\") " pod="openshift-dns/node-resolver-fqtzs"
Oct 30 00:31:11.922972 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.922948    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ef9a0a39-051f-45a1-acc9-354a23e5b00b-system-cni-dir\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:11.922972 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.922970    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05a614f5-3112-4216-b894-68bf3945458c-tmp-dir\") pod \"node-resolver-fqtzs\" (UID: \"05a614f5-3112-4216-b894-68bf3945458c\") " pod="openshift-dns/node-resolver-fqtzs"
Oct 30 00:31:11.923069 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.922986    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-run-netns\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.923069 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.923000    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-cni-bin\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.923069 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.923023    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.923069 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.923040    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-system-cni-dir\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.923069 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.923057    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xvts\" (UniqueName: \"kubernetes.io/projected/b3705749-d093-4b49-878d-e4616e81a450-kube-api-access-7xvts\") pod \"network-check-target-wpvjg\" (UID: \"b3705749-d093-4b49-878d-e4616e81a450\") " pod="openshift-network-diagnostics/network-check-target-wpvjg"
Oct 30 00:31:11.923222 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.923072    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-sys-fs\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:11.923222 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.923088    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30e43c48-92a7-4c68-887c-9aaf7d6d1e1b-host\") pod \"node-ca-9hslh\" (UID: \"30e43c48-92a7-4c68-887c-9aaf7d6d1e1b\") " pod="openshift-image-registry/node-ca-9hslh"
Oct 30 00:31:11.923222 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.923104    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ef9a0a39-051f-45a1-acc9-354a23e5b00b-cni-binary-copy\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:11.923222 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.923127    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ef9a0a39-051f-45a1-acc9-354a23e5b00b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:11.923222 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.923144    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pchtx\" (UniqueName: \"kubernetes.io/projected/05a614f5-3112-4216-b894-68bf3945458c-kube-api-access-pchtx\") pod \"node-resolver-fqtzs\" (UID: \"05a614f5-3112-4216-b894-68bf3945458c\") " pod="openshift-dns/node-resolver-fqtzs"
Oct 30 00:31:11.923222 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.923189    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e552ba30-5b09-48a7-bffb-6a65a094c2bc-metrics-certs\") pod \"network-metrics-daemon-jldmt\" (UID: \"e552ba30-5b09-48a7-bffb-6a65a094c2bc\") " pod="openshift-multus/network-metrics-daemon-jldmt"
Oct 30 00:31:11.923222 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.923188    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4a99683d-3f83-465d-9152-2f422641e087-iptables-alerter-script\") pod \"iptables-alerter-s72l8\" (UID: \"4a99683d-3f83-465d-9152-2f422641e087\") " pod="openshift-network-operator/iptables-alerter-s72l8"
Oct 30 00:31:11.923222 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.923204    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-cni-netd\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.923222 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.923221    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/30e43c48-92a7-4c68-887c-9aaf7d6d1e1b-serviceca\") pod \"node-ca-9hslh\" (UID: \"30e43c48-92a7-4c68-887c-9aaf7d6d1e1b\") " pod="openshift-image-registry/node-ca-9hslh"
Oct 30 00:31:11.923494 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.923236    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-host\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.923534 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.923514    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-sys-fs\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:11.923626 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.923602    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/30e43c48-92a7-4c68-887c-9aaf7d6d1e1b-serviceca\") pod \"node-ca-9hslh\" (UID: \"30e43c48-92a7-4c68-887c-9aaf7d6d1e1b\") " pod="openshift-image-registry/node-ca-9hslh"
Oct 30 00:31:11.923732 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.923645    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-cni-bin\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.923732 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.923664    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/05a614f5-3112-4216-b894-68bf3945458c-hosts-file\") pod \"node-resolver-fqtzs\" (UID: \"05a614f5-3112-4216-b894-68bf3945458c\") " pod="openshift-dns/node-resolver-fqtzs"
Oct 30 00:31:11.923732 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.923671    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30e43c48-92a7-4c68-887c-9aaf7d6d1e1b-host\") pod \"node-ca-9hslh\" (UID: \"30e43c48-92a7-4c68-887c-9aaf7d6d1e1b\") " pod="openshift-image-registry/node-ca-9hslh"
Oct 30 00:31:11.923732 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.923688    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-device-dir\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:11.923732 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.923701    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.923732 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.923705    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpfcq\" (UniqueName: \"kubernetes.io/projected/e552ba30-5b09-48a7-bffb-6a65a094c2bc-kube-api-access-gpfcq\") pod \"network-metrics-daemon-jldmt\" (UID: \"e552ba30-5b09-48a7-bffb-6a65a094c2bc\") " pod="openshift-multus/network-metrics-daemon-jldmt"
Oct 30 00:31:11.923985 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.923735    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-systemd-units\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.923985 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.923760    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-slash\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.923985 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.923801    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-host-var-lib-cni-bin\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.923985 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.923820    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-multus-conf-dir\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.923985 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.923849    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-etc-modprobe-d\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.923985 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.923877    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-etc-systemd\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.923985 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.923940    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ef9a0a39-051f-45a1-acc9-354a23e5b00b-system-cni-dir\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:11.924236 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.924014    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-etc-modprobe-d\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.924236 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:11.924092    2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:31:11.924236 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:11.924165    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e552ba30-5b09-48a7-bffb-6a65a094c2bc-metrics-certs podName:e552ba30-5b09-48a7-bffb-6a65a094c2bc nodeName:}" failed. No retries permitted until 2026-10-30 00:31:12.424148064 +0000 UTC m=+3.050963893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e552ba30-5b09-48a7-bffb-6a65a094c2bc-metrics-certs") pod "network-metrics-daemon-jldmt" (UID: "e552ba30-5b09-48a7-bffb-6a65a094c2bc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:31:11.924236 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.924198    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05a614f5-3112-4216-b894-68bf3945458c-tmp-dir\") pod \"node-resolver-fqtzs\" (UID: \"05a614f5-3112-4216-b894-68bf3945458c\") " pod="openshift-dns/node-resolver-fqtzs"
Oct 30 00:31:11.924393 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.924237    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-cni-netd\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.924431 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.924396    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-host\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.924464 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.924430    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-run-netns\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.924464 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.924459    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-device-dir\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:11.924555 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.924492    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-systemd-units\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.924555 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.924537    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-system-cni-dir\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.924722 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.924705    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ef9a0a39-051f-45a1-acc9-354a23e5b00b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:11.924802 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.924749    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a99683d-3f83-465d-9152-2f422641e087-host-slash\") pod \"iptables-alerter-s72l8\" (UID: \"4a99683d-3f83-465d-9152-2f422641e087\") " pod="openshift-network-operator/iptables-alerter-s72l8"
Oct 30 00:31:11.924802 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.924767    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:11.924895 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.924817    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-etc-openvswitch\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.924895 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.924846    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-ovnkube-config\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.924962 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.924916    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-multus-socket-dir-parent\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.924962 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.924929    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ef9a0a39-051f-45a1-acc9-354a23e5b00b-cni-binary-copy\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:11.925045 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.924977    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-etc-systemd\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.925045 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.924984    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-etc-sysconfig\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.925045 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925013    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-slash\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.925045 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925015    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-host-var-lib-cni-bin\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.925045 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925039    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-multus-conf-dir\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.925199 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925049    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a99683d-3f83-465d-9152-2f422641e087-host-slash\") pod \"iptables-alerter-s72l8\" (UID: \"4a99683d-3f83-465d-9152-2f422641e087\") " pod="openshift-network-operator/iptables-alerter-s72l8"
Oct 30 00:31:11.925199 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.924935    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-etc-sysconfig\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.925199 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925079    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-sys\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.925199 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925095    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bgt8\" (UniqueName: \"kubernetes.io/projected/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-kube-api-access-4bgt8\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:11.925199 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925111    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-run-ovn-kubernetes\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.925199 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925126    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-host-run-k8s-cni-cncf-io\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.925199 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925178    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-host-run-netns\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.925440 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925203    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5687019d-945a-4307-8269-5c0ae6f377ec-tmp\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.925440 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925227    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-socket-dir\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:11.925440 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925249    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-registration-dir\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:11.925440 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925267    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ef9a0a39-051f-45a1-acc9-354a23e5b00b-cnibin\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:11.925440 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925285    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3b0df054-0749-4417-bb8b-b12b3353d25e-kubelet-config\") pod \"global-pull-secret-syncer-f27vk\" (UID: \"3b0df054-0749-4417-bb8b-b12b3353d25e\") " pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:11.925440 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925313    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b0df054-0749-4417-bb8b-b12b3353d25e-original-pull-secret\") pod \"global-pull-secret-syncer-f27vk\" (UID: \"3b0df054-0749-4417-bb8b-b12b3353d25e\") " pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:11.925440 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925336    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-etc-sysctl-d\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.925678 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925440    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-node-log\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.925678 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925463    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-env-overrides\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.925678 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925464    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-ovnkube-config\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.925678 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925479    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-ovnkube-script-lib\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.925678 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925504    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-host-run-netns\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.925678 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925528    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3b0df054-0749-4417-bb8b-b12b3353d25e-dbus\") pod \"global-pull-secret-syncer-f27vk\" (UID: \"3b0df054-0749-4417-bb8b-b12b3353d25e\") " pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:11.925678 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925557    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-cnibin\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.925678 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925573    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-host-var-lib-cni-multus\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.925678 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925593    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-host-run-multus-certs\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.925678 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925610    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5687019d-945a-4307-8269-5c0ae6f377ec-etc-tuned\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.925678 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925626    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4f7ca654-edeb-4457-9239-8da741b5dafd-konnectivity-ca\") pod \"konnectivity-agent-6wzd9\" (UID: \"4f7ca654-edeb-4457-9239-8da741b5dafd\") " pod="kube-system/konnectivity-agent-6wzd9"
Oct 30 00:31:11.925678 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925642    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28lgt\" (UniqueName: \"kubernetes.io/projected/4a99683d-3f83-465d-9152-2f422641e087-kube-api-access-28lgt\") pod \"iptables-alerter-s72l8\" (UID: \"4a99683d-3f83-465d-9152-2f422641e087\") " pod="openshift-network-operator/iptables-alerter-s72l8"
Oct 30 00:31:11.925678 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925657    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-etc-selinux\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:11.925678 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925673    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-kubelet\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.926264 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925697    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-log-socket\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.926264 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925728    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-run-ovn-kubernetes\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.926264 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925805    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-host-run-k8s-cni-cncf-io\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.926264 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925822    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-host-run-multus-certs\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.926264 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925854    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:11.926264 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925868    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-etc-sysctl-d\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.926264 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925945    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-multus-socket-dir-parent\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.926264 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925956    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3b0df054-0749-4417-bb8b-b12b3353d25e-dbus\") pod \"global-pull-secret-syncer-f27vk\" (UID: \"3b0df054-0749-4417-bb8b-b12b3353d25e\") " pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:11.926264 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.925995    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ef9a0a39-051f-45a1-acc9-354a23e5b00b-cnibin\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:11.926264 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.926021    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-kubelet\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.926264 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.926075    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wblgx\" (UniqueName: \"kubernetes.io/projected/ef9a0a39-051f-45a1-acc9-354a23e5b00b-kube-api-access-wblgx\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:11.926264 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.926096    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3b734765-8e7f-4d8e-a74b-bd179adf33ee-multus-daemon-config\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.926264 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.926120    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4f7ca654-edeb-4457-9239-8da741b5dafd-agent-certs\") pod \"konnectivity-agent-6wzd9\" (UID: \"4f7ca654-edeb-4457-9239-8da741b5dafd\") " pod="kube-system/konnectivity-agent-6wzd9"
Oct 30 00:31:11.926264 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.926142    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-run-systemd\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.926264 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.926171    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-etc-kubernetes\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.926264 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.926196    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-lib-modules\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.926264 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.926221    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqzmf\" (UniqueName: \"kubernetes.io/projected/5687019d-945a-4307-8269-5c0ae6f377ec-kube-api-access-gqzmf\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.927087 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.926254    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-var-lib-openvswitch\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.927087 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.926283    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgw94\" (UniqueName: \"kubernetes.io/projected/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-kube-api-access-bgw94\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.927087 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.926344    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-var-lib-openvswitch\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.927087 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.926350    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-run-systemd\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.927087 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.926361    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-cnibin\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.927087 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.926387    2548 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 00:31:11.927087 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.926404    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-etc-kubernetes\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.927087 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.926437    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-ovnkube-script-lib\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.927087 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.926446    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-run\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.927087 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.926451    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-socket-dir\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:11.927087 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.926476    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ef9a0a39-051f-45a1-acc9-354a23e5b00b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:11.927087 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.926488    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-node-log\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.927087 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.926506    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-multus-cni-dir\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.927087 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.926562    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-multus-cni-dir\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.927087 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.926589    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-etc-openvswitch\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.927087 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.926613    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-run\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.927087 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.926626    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4f7ca654-edeb-4457-9239-8da741b5dafd-konnectivity-ca\") pod \"konnectivity-agent-6wzd9\" (UID: \"4f7ca654-edeb-4457-9239-8da741b5dafd\") " pod="kube-system/konnectivity-agent-6wzd9"
Oct 30 00:31:11.927087 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.926686    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3b0df054-0749-4417-bb8b-b12b3353d25e-kubelet-config\") pod \"global-pull-secret-syncer-f27vk\" (UID: \"3b0df054-0749-4417-bb8b-b12b3353d25e\") " pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:11.927725 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.926729    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-registration-dir\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:11.927725 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.926732    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ef9a0a39-051f-45a1-acc9-354a23e5b00b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:11.927725 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.926816    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-env-overrides\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.927725 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:11.926818    2548 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:31:11.927725 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:11.926888    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b0df054-0749-4417-bb8b-b12b3353d25e-original-pull-secret podName:3b0df054-0749-4417-bb8b-b12b3353d25e nodeName:}" failed. No retries permitted until 2026-10-30 00:31:12.426869752 +0000 UTC m=+3.053685590 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b0df054-0749-4417-bb8b-b12b3353d25e-original-pull-secret") pod "global-pull-secret-syncer-f27vk" (UID: "3b0df054-0749-4417-bb8b-b12b3353d25e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:31:11.927725 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.926914    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-lib-modules\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.927725 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.926929    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzcz2\" (UniqueName: \"kubernetes.io/projected/30e43c48-92a7-4c68-887c-9aaf7d6d1e1b-kube-api-access-zzcz2\") pod \"node-ca-9hslh\" (UID: \"30e43c48-92a7-4c68-887c-9aaf7d6d1e1b\") " pod="openshift-image-registry/node-ca-9hslh"
Oct 30 00:31:11.927725 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.926957    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-etc-selinux\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:11.927725 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.927012    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-host-var-lib-cni-multus\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.927725 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.927032    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-log-socket\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.927725 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.927055    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-sys\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.927725 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.927188    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3b734765-8e7f-4d8e-a74b-bd179adf33ee-multus-daemon-config\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.927725 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.927333    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ef9a0a39-051f-45a1-acc9-354a23e5b00b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:11.927725 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.927394    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7rhg\" (UniqueName: \"kubernetes.io/projected/3b734765-8e7f-4d8e-a74b-bd179adf33ee-kube-api-access-k7rhg\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.927725 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.927426    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-etc-kubernetes\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.927725 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.927465    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-ovn-node-metrics-cert\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.927725 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.927490    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-hostroot\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.928428 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.927514    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-var-lib-kubelet\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.928428 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.927531    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-run-openvswitch\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.928428 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.927567    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-run-ovn\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.928428 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.927583    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b734765-8e7f-4d8e-a74b-bd179adf33ee-cni-binary-copy\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.928428 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.927598    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-host-var-lib-kubelet\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.928428 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.927668    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-host-var-lib-kubelet\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.928428 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.927705    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-run-openvswitch\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.928428 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.927750    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-etc-kubernetes\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.928428 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.927838    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-hostroot\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.928428 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.928044    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-run-ovn\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.928428 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.928115    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-var-lib-kubelet\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.928428 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.928217    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ef9a0a39-051f-45a1-acc9-354a23e5b00b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:11.928883 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.928529    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b734765-8e7f-4d8e-a74b-bd179adf33ee-cni-binary-copy\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:11.930424 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.930401    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5687019d-945a-4307-8269-5c0ae6f377ec-tmp\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.930688 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.930660    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4f7ca654-edeb-4457-9239-8da741b5dafd-agent-certs\") pod \"konnectivity-agent-6wzd9\" (UID: \"4f7ca654-edeb-4457-9239-8da741b5dafd\") " pod="kube-system/konnectivity-agent-6wzd9"
Oct 30 00:31:11.931212 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.931187    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-ovn-node-metrics-cert\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.931407 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.931354    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5687019d-945a-4307-8269-5c0ae6f377ec-etc-tuned\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.942340 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:11.942308    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:31:11.942340 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:11.942346    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:31:11.942592 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:11.942361    2548 projected.go:196] Error preparing data for projected volume kube-api-access-7xvts for pod openshift-network-diagnostics/network-check-target-wpvjg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:31:11.942592 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:11.942453    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3705749-d093-4b49-878d-e4616e81a450-kube-api-access-7xvts podName:b3705749-d093-4b49-878d-e4616e81a450 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:12.442433014 +0000 UTC m=+3.069248834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7xvts" (UniqueName: "kubernetes.io/projected/b3705749-d093-4b49-878d-e4616e81a450-kube-api-access-7xvts") pod "network-check-target-wpvjg" (UID: "b3705749-d093-4b49-878d-e4616e81a450") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:31:11.946497 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.946411    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpfcq\" (UniqueName: \"kubernetes.io/projected/e552ba30-5b09-48a7-bffb-6a65a094c2bc-kube-api-access-gpfcq\") pod \"network-metrics-daemon-jldmt\" (UID: \"e552ba30-5b09-48a7-bffb-6a65a094c2bc\") " pod="openshift-multus/network-metrics-daemon-jldmt"
Oct 30 00:31:11.947011 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.946984    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqzmf\" (UniqueName: \"kubernetes.io/projected/5687019d-945a-4307-8269-5c0ae6f377ec-kube-api-access-gqzmf\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:11.947812 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.947765    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28lgt\" (UniqueName: \"kubernetes.io/projected/4a99683d-3f83-465d-9152-2f422641e087-kube-api-access-28lgt\") pod \"iptables-alerter-s72l8\" (UID: \"4a99683d-3f83-465d-9152-2f422641e087\") " pod="openshift-network-operator/iptables-alerter-s72l8"
Oct 30 00:31:11.948714 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.948690    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wblgx\" (UniqueName: \"kubernetes.io/projected/ef9a0a39-051f-45a1-acc9-354a23e5b00b-kube-api-access-wblgx\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:11.949209 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.949184    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgw94\" (UniqueName: \"kubernetes.io/projected/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-kube-api-access-bgw94\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:11.950495 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.950472    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pchtx\" (UniqueName: \"kubernetes.io/projected/05a614f5-3112-4216-b894-68bf3945458c-kube-api-access-pchtx\") pod \"node-resolver-fqtzs\" (UID: \"05a614f5-3112-4216-b894-68bf3945458c\") " pod="openshift-dns/node-resolver-fqtzs"
Oct 30 00:31:11.957551 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.957523    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzcz2\" (UniqueName: \"kubernetes.io/projected/30e43c48-92a7-4c68-887c-9aaf7d6d1e1b-kube-api-access-zzcz2\") pod \"node-ca-9hslh\" (UID: \"30e43c48-92a7-4c68-887c-9aaf7d6d1e1b\") " pod="openshift-image-registry/node-ca-9hslh"
Oct 30 00:31:11.957668 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.957556    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bgt8\" (UniqueName: \"kubernetes.io/projected/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-kube-api-access-4bgt8\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:11.959555 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:11.959534    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7rhg\" (UniqueName: \"kubernetes.io/projected/3b734765-8e7f-4d8e-a74b-bd179adf33ee-kube-api-access-k7rhg\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:12.052994 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:12.052920    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:12.055771 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:12.055742    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9hslh"
Oct 30 00:31:12.062583 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:12.062551    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r58sp"
Oct 30 00:31:12.064966 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:12.064938    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:12.065077 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:12.064931    2548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bdb8d7d_1c89_49ae_ab3b_5b7bbe2c1835.slice/crio-08b687a874a3fd85b6a79f5d3e1ed5360f420760e272fdcc41b9ffd9e44ef64e WatchSource:0}: Error finding container 08b687a874a3fd85b6a79f5d3e1ed5360f420760e272fdcc41b9ffd9e44ef64e: Status 404 returned error can't find the container with id 08b687a874a3fd85b6a79f5d3e1ed5360f420760e272fdcc41b9ffd9e44ef64e
Oct 30 00:31:12.068837 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:12.068811    2548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30e43c48_92a7_4c68_887c_9aaf7d6d1e1b.slice/crio-30d33f546b4ee2cdbc14239fa2b91763e67dabbe72208cdf284d7622473fdf31 WatchSource:0}: Error finding container 30d33f546b4ee2cdbc14239fa2b91763e67dabbe72208cdf284d7622473fdf31: Status 404 returned error can't find the container with id 30d33f546b4ee2cdbc14239fa2b91763e67dabbe72208cdf284d7622473fdf31
Oct 30 00:31:12.072954 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:12.072935    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6wzd9"
Oct 30 00:31:12.082252 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:12.082073    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s72l8"
Oct 30 00:31:12.090129 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:12.090078    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fqtzs"
Oct 30 00:31:12.103621 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:12.103600    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:12.105455 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:12.105423    2548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b734765_8e7f_4d8e_a74b_bd179adf33ee.slice/crio-711a65b4aa0b4b6a781f42961ea6978c5f8abc36884897ab61372cd9bd07c1d1 WatchSource:0}: Error finding container 711a65b4aa0b4b6a781f42961ea6978c5f8abc36884897ab61372cd9bd07c1d1: Status 404 returned error can't find the container with id 711a65b4aa0b4b6a781f42961ea6978c5f8abc36884897ab61372cd9bd07c1d1
Oct 30 00:31:12.107084 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:12.106988    2548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5687019d_945a_4307_8269_5c0ae6f377ec.slice/crio-04cb0c16ee0c2411462c975b4e45bf147db4fcf6b03392b7c159f405a872a2ff WatchSource:0}: Error finding container 04cb0c16ee0c2411462c975b4e45bf147db4fcf6b03392b7c159f405a872a2ff: Status 404 returned error can't find the container with id 04cb0c16ee0c2411462c975b4e45bf147db4fcf6b03392b7c159f405a872a2ff
Oct 30 00:31:12.110635 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:12.110609    2548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a99683d_3f83_465d_9152_2f422641e087.slice/crio-9a00488f5b29d78963136584d8f9efc0625fce53066cf2967bddb4263ed92262 WatchSource:0}: Error finding container 9a00488f5b29d78963136584d8f9efc0625fce53066cf2967bddb4263ed92262: Status 404 returned error can't find the container with id 9a00488f5b29d78963136584d8f9efc0625fce53066cf2967bddb4263ed92262
Oct 30 00:31:12.112610 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:12.112580    2548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f7ca654_edeb_4457_9239_8da741b5dafd.slice/crio-02f558af7f3ca19844c85f4ca8a14e68f33a793e7f1626b2b5140ebbea253e6e WatchSource:0}: Error finding container 02f558af7f3ca19844c85f4ca8a14e68f33a793e7f1626b2b5140ebbea253e6e: Status 404 returned error can't find the container with id 02f558af7f3ca19844c85f4ca8a14e68f33a793e7f1626b2b5140ebbea253e6e
Oct 30 00:31:12.117134 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:12.117112    2548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05a614f5_3112_4216_b894_68bf3945458c.slice/crio-2891385a68bce7a18baf0301755c0df93db5e2fe2a59bdcb82a33771fb497327 WatchSource:0}: Error finding container 2891385a68bce7a18baf0301755c0df93db5e2fe2a59bdcb82a33771fb497327: Status 404 returned error can't find the container with id 2891385a68bce7a18baf0301755c0df93db5e2fe2a59bdcb82a33771fb497327
Oct 30 00:31:12.118276 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:12.118251    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:12.120536 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:12.120439    2548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b48e29_13f0_4f4d_a19c_ddaa7e2431dc.slice/crio-7f882d988379e6b5bccfffdc83df6afc60d627bab3911f201a5e438a435c62a6 WatchSource:0}: Error finding container 7f882d988379e6b5bccfffdc83df6afc60d627bab3911f201a5e438a435c62a6: Status 404 returned error can't find the container with id 7f882d988379e6b5bccfffdc83df6afc60d627bab3911f201a5e438a435c62a6
Oct 30 00:31:12.132069 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:12.132044    2548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef9a0a39_051f_45a1_acc9_354a23e5b00b.slice/crio-7f08e3175a14b08ecca39bbea8f08cfdcbe330df1f4544751cf218dc6e2c11bd WatchSource:0}: Error finding container 7f08e3175a14b08ecca39bbea8f08cfdcbe330df1f4544751cf218dc6e2c11bd: Status 404 returned error can't find the container with id 7f08e3175a14b08ecca39bbea8f08cfdcbe330df1f4544751cf218dc6e2c11bd
Oct 30 00:31:12.432946 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:12.432653    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e552ba30-5b09-48a7-bffb-6a65a094c2bc-metrics-certs\") pod \"network-metrics-daemon-jldmt\" (UID: \"e552ba30-5b09-48a7-bffb-6a65a094c2bc\") " pod="openshift-multus/network-metrics-daemon-jldmt"
Oct 30 00:31:12.433135 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:12.432987    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b0df054-0749-4417-bb8b-b12b3353d25e-original-pull-secret\") pod \"global-pull-secret-syncer-f27vk\" (UID: \"3b0df054-0749-4417-bb8b-b12b3353d25e\") " pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:12.433135 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:12.432837    2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:31:12.433135 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:12.433092    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e552ba30-5b09-48a7-bffb-6a65a094c2bc-metrics-certs podName:e552ba30-5b09-48a7-bffb-6a65a094c2bc nodeName:}" failed. No retries permitted until 2026-10-30 00:31:13.433068862 +0000 UTC m=+4.059884680 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e552ba30-5b09-48a7-bffb-6a65a094c2bc-metrics-certs") pod "network-metrics-daemon-jldmt" (UID: "e552ba30-5b09-48a7-bffb-6a65a094c2bc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:31:12.433135 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:12.433127    2548 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:31:12.433343 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:12.433178    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b0df054-0749-4417-bb8b-b12b3353d25e-original-pull-secret podName:3b0df054-0749-4417-bb8b-b12b3353d25e nodeName:}" failed. No retries permitted until 2026-10-30 00:31:13.433163433 +0000 UTC m=+4.059979240 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b0df054-0749-4417-bb8b-b12b3353d25e-original-pull-secret") pod "global-pull-secret-syncer-f27vk" (UID: "3b0df054-0749-4417-bb8b-b12b3353d25e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:31:12.534057 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:12.534014    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xvts\" (UniqueName: \"kubernetes.io/projected/b3705749-d093-4b49-878d-e4616e81a450-kube-api-access-7xvts\") pod \"network-check-target-wpvjg\" (UID: \"b3705749-d093-4b49-878d-e4616e81a450\") " pod="openshift-network-diagnostics/network-check-target-wpvjg"
Oct 30 00:31:12.534262 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:12.534199    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:31:12.534262 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:12.534222    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:31:12.534262 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:12.534234    2548 projected.go:196] Error preparing data for projected volume kube-api-access-7xvts for pod openshift-network-diagnostics/network-check-target-wpvjg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:31:12.534421 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:12.534297    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3705749-d093-4b49-878d-e4616e81a450-kube-api-access-7xvts podName:b3705749-d093-4b49-878d-e4616e81a450 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:13.534277751 +0000 UTC m=+4.161093575 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7xvts" (UniqueName: "kubernetes.io/projected/b3705749-d093-4b49-878d-e4616e81a450-kube-api-access-7xvts") pod "network-check-target-wpvjg" (UID: "b3705749-d093-4b49-878d-e4616e81a450") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:31:12.763618 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:12.762763    2548 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:26:11 +0000 UTC" deadline="2028-04-24 13:24:19.985334761 +0000 UTC"
Oct 30 00:31:12.763618 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:12.762857    2548 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13020h53m7.22248289s"
Oct 30 00:31:12.818663 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:12.818618    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-56.ec2.internal" event={"ID":"21539217511a9b702eb77d59ab0b339d","Type":"ContainerStarted","Data":"db9b07011d2f050f681fa96d04710e99dfd894cfae442af0b42780ee1d573d36"}
Oct 30 00:31:12.833901 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:12.833865    2548 generic.go:356] "Generic (PLEG): container finished" podID="f3ebcfeede45130240ea614a61281a18" containerID="37ac9b4631a778e3a0b9dfa9c14f01a951ec5805b9833c8d20097088bf7fba89" exitCode=0
Oct 30 00:31:12.834076 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:12.833976    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal" event={"ID":"f3ebcfeede45130240ea614a61281a18","Type":"ContainerDied","Data":"37ac9b4631a778e3a0b9dfa9c14f01a951ec5805b9833c8d20097088bf7fba89"}
Oct 30 00:31:12.837002 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:12.836971    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz" event={"ID":"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc","Type":"ContainerStarted","Data":"7f882d988379e6b5bccfffdc83df6afc60d627bab3911f201a5e438a435c62a6"}
Oct 30 00:31:12.848395 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:12.848360    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fqtzs" event={"ID":"05a614f5-3112-4216-b894-68bf3945458c","Type":"ContainerStarted","Data":"2891385a68bce7a18baf0301755c0df93db5e2fe2a59bdcb82a33771fb497327"}
Oct 30 00:31:12.867491 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:12.867326    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lnnlc" event={"ID":"5687019d-945a-4307-8269-5c0ae6f377ec","Type":"ContainerStarted","Data":"04cb0c16ee0c2411462c975b4e45bf147db4fcf6b03392b7c159f405a872a2ff"}
Oct 30 00:31:12.875106 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:12.875077    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r58sp" event={"ID":"3b734765-8e7f-4d8e-a74b-bd179adf33ee","Type":"ContainerStarted","Data":"711a65b4aa0b4b6a781f42961ea6978c5f8abc36884897ab61372cd9bd07c1d1"}
Oct 30 00:31:12.884986 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:12.884914    2548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-56.ec2.internal" podStartSLOduration=2.884894598 podStartE2EDuration="2.884894598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:31:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:12.849926362 +0000 UTC m=+3.476742190" watchObservedRunningTime="2026-10-30 00:31:12.884894598 +0000 UTC m=+3.511710425"
Oct 30 00:31:12.898683 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:12.898648    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9hslh" event={"ID":"30e43c48-92a7-4c68-887c-9aaf7d6d1e1b","Type":"ContainerStarted","Data":"30d33f546b4ee2cdbc14239fa2b91763e67dabbe72208cdf284d7622473fdf31"}
Oct 30 00:31:12.905194 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:12.905162    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp" event={"ID":"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835","Type":"ContainerStarted","Data":"08b687a874a3fd85b6a79f5d3e1ed5360f420760e272fdcc41b9ffd9e44ef64e"}
Oct 30 00:31:12.929736 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:12.929698    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-csc84" event={"ID":"ef9a0a39-051f-45a1-acc9-354a23e5b00b","Type":"ContainerStarted","Data":"7f08e3175a14b08ecca39bbea8f08cfdcbe330df1f4544751cf218dc6e2c11bd"}
Oct 30 00:31:12.941103 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:12.941010    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6wzd9" event={"ID":"4f7ca654-edeb-4457-9239-8da741b5dafd","Type":"ContainerStarted","Data":"02f558af7f3ca19844c85f4ca8a14e68f33a793e7f1626b2b5140ebbea253e6e"}
Oct 30 00:31:12.969832 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:12.969675    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s72l8" event={"ID":"4a99683d-3f83-465d-9152-2f422641e087","Type":"ContainerStarted","Data":"9a00488f5b29d78963136584d8f9efc0625fce53066cf2967bddb4263ed92262"}
Oct 30 00:31:13.440829 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:13.440788    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b0df054-0749-4417-bb8b-b12b3353d25e-original-pull-secret\") pod \"global-pull-secret-syncer-f27vk\" (UID: \"3b0df054-0749-4417-bb8b-b12b3353d25e\") " pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:13.441009 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:13.440884    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e552ba30-5b09-48a7-bffb-6a65a094c2bc-metrics-certs\") pod \"network-metrics-daemon-jldmt\" (UID: \"e552ba30-5b09-48a7-bffb-6a65a094c2bc\") " pod="openshift-multus/network-metrics-daemon-jldmt"
Oct 30 00:31:13.441092 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:13.441013    2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:31:13.441092 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:13.441077    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e552ba30-5b09-48a7-bffb-6a65a094c2bc-metrics-certs podName:e552ba30-5b09-48a7-bffb-6a65a094c2bc nodeName:}" failed. No retries permitted until 2026-10-30 00:31:15.441058129 +0000 UTC m=+6.067873952 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e552ba30-5b09-48a7-bffb-6a65a094c2bc-metrics-certs") pod "network-metrics-daemon-jldmt" (UID: "e552ba30-5b09-48a7-bffb-6a65a094c2bc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:31:13.441202 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:13.441140    2548 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:31:13.441202 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:13.441188    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b0df054-0749-4417-bb8b-b12b3353d25e-original-pull-secret podName:3b0df054-0749-4417-bb8b-b12b3353d25e nodeName:}" failed. No retries permitted until 2026-10-30 00:31:15.441177426 +0000 UTC m=+6.067993235 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b0df054-0749-4417-bb8b-b12b3353d25e-original-pull-secret") pod "global-pull-secret-syncer-f27vk" (UID: "3b0df054-0749-4417-bb8b-b12b3353d25e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:31:13.541611 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:13.541536    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xvts\" (UniqueName: \"kubernetes.io/projected/b3705749-d093-4b49-878d-e4616e81a450-kube-api-access-7xvts\") pod \"network-check-target-wpvjg\" (UID: \"b3705749-d093-4b49-878d-e4616e81a450\") " pod="openshift-network-diagnostics/network-check-target-wpvjg"
Oct 30 00:31:13.541826 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:13.541753    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:31:13.541826 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:13.541774    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:31:13.541826 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:13.541801    2548 projected.go:196] Error preparing data for projected volume kube-api-access-7xvts for pod openshift-network-diagnostics/network-check-target-wpvjg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:31:13.541982 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:13.541859    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3705749-d093-4b49-878d-e4616e81a450-kube-api-access-7xvts podName:b3705749-d093-4b49-878d-e4616e81a450 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:15.541841061 +0000 UTC m=+6.168657056 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7xvts" (UniqueName: "kubernetes.io/projected/b3705749-d093-4b49-878d-e4616e81a450-kube-api-access-7xvts") pod "network-check-target-wpvjg" (UID: "b3705749-d093-4b49-878d-e4616e81a450") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:31:13.764349 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:13.764257    2548 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:26:11 +0000 UTC" deadline="2028-04-24 12:06:43.147232398 +0000 UTC"
Oct 30 00:31:13.764349 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:13.764305    2548 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13019h35m29.382932729s"
Oct 30 00:31:13.781983 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:13.781947    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpvjg"
Oct 30 00:31:13.782162 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:13.782086    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpvjg" podUID="b3705749-d093-4b49-878d-e4616e81a450"
Oct 30 00:31:13.782531 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:13.782504    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jldmt"
Oct 30 00:31:13.782667 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:13.782636    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jldmt" podUID="e552ba30-5b09-48a7-bffb-6a65a094c2bc"
Oct 30 00:31:13.782761 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:13.782705    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:13.782881 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:13.782771    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f27vk" podUID="3b0df054-0749-4417-bb8b-b12b3353d25e"
Oct 30 00:31:13.995955 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:13.995851    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal" event={"ID":"f3ebcfeede45130240ea614a61281a18","Type":"ContainerStarted","Data":"3d15129431a215893a9646f835eb4259905f2df4159fe7794fc9182186624040"}
Oct 30 00:31:15.458006 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:15.457954    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e552ba30-5b09-48a7-bffb-6a65a094c2bc-metrics-certs\") pod \"network-metrics-daemon-jldmt\" (UID: \"e552ba30-5b09-48a7-bffb-6a65a094c2bc\") " pod="openshift-multus/network-metrics-daemon-jldmt"
Oct 30 00:31:15.458502 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:15.458023    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b0df054-0749-4417-bb8b-b12b3353d25e-original-pull-secret\") pod \"global-pull-secret-syncer-f27vk\" (UID: \"3b0df054-0749-4417-bb8b-b12b3353d25e\") " pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:15.458502 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:15.458162    2548 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:31:15.458502 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:15.458224    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b0df054-0749-4417-bb8b-b12b3353d25e-original-pull-secret podName:3b0df054-0749-4417-bb8b-b12b3353d25e nodeName:}" failed. No retries permitted until 2026-10-30 00:31:19.458206676 +0000 UTC m=+10.085022482 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b0df054-0749-4417-bb8b-b12b3353d25e-original-pull-secret") pod "global-pull-secret-syncer-f27vk" (UID: "3b0df054-0749-4417-bb8b-b12b3353d25e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:31:15.458666 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:15.458632    2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:31:15.458712 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:15.458685    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e552ba30-5b09-48a7-bffb-6a65a094c2bc-metrics-certs podName:e552ba30-5b09-48a7-bffb-6a65a094c2bc nodeName:}" failed. No retries permitted until 2026-10-30 00:31:19.45866877 +0000 UTC m=+10.085484588 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e552ba30-5b09-48a7-bffb-6a65a094c2bc-metrics-certs") pod "network-metrics-daemon-jldmt" (UID: "e552ba30-5b09-48a7-bffb-6a65a094c2bc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:31:15.559870 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:15.559173    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xvts\" (UniqueName: \"kubernetes.io/projected/b3705749-d093-4b49-878d-e4616e81a450-kube-api-access-7xvts\") pod \"network-check-target-wpvjg\" (UID: \"b3705749-d093-4b49-878d-e4616e81a450\") " pod="openshift-network-diagnostics/network-check-target-wpvjg"
Oct 30 00:31:15.559870 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:15.559389    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:31:15.559870 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:15.559413    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:31:15.559870 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:15.559427    2548 projected.go:196] Error preparing data for projected volume kube-api-access-7xvts for pod openshift-network-diagnostics/network-check-target-wpvjg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:31:15.559870 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:15.559488    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3705749-d093-4b49-878d-e4616e81a450-kube-api-access-7xvts podName:b3705749-d093-4b49-878d-e4616e81a450 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:19.559469548 +0000 UTC m=+10.186285374 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7xvts" (UniqueName: "kubernetes.io/projected/b3705749-d093-4b49-878d-e4616e81a450-kube-api-access-7xvts") pod "network-check-target-wpvjg" (UID: "b3705749-d093-4b49-878d-e4616e81a450") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:31:15.781850 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:15.781746    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpvjg"
Oct 30 00:31:15.782000 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:15.781895    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpvjg" podUID="b3705749-d093-4b49-878d-e4616e81a450"
Oct 30 00:31:15.782346 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:15.782320    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jldmt"
Oct 30 00:31:15.782512 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:15.782456    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jldmt" podUID="e552ba30-5b09-48a7-bffb-6a65a094c2bc"
Oct 30 00:31:15.782597 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:15.782528    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:15.782650 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:15.782611    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f27vk" podUID="3b0df054-0749-4417-bb8b-b12b3353d25e"
Oct 30 00:31:16.012494 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:16.012455    2548 generic.go:356] "Generic (PLEG): container finished" podID="ef9a0a39-051f-45a1-acc9-354a23e5b00b" containerID="8fb55fece667206aca4113a3508bf1902142200d36f6997471005c9313e00ad0" exitCode=0
Oct 30 00:31:16.012634 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:16.012523    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-csc84" event={"ID":"ef9a0a39-051f-45a1-acc9-354a23e5b00b","Type":"ContainerDied","Data":"8fb55fece667206aca4113a3508bf1902142200d36f6997471005c9313e00ad0"}
Oct 30 00:31:16.067741 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:16.067561    2548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal" podStartSLOduration=6.067543673 podStartE2EDuration="6.067543673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:31:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:14.028543838 +0000 UTC m=+4.655359665" watchObservedRunningTime="2026-10-30 00:31:16.067543673 +0000 UTC m=+6.694359500"
Oct 30 00:31:17.023461 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:17.022804    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6wzd9" event={"ID":"4f7ca654-edeb-4457-9239-8da741b5dafd","Type":"ContainerStarted","Data":"68aa51750bc21a707fdac214924c8defcf2578ea127b77a14f3072adfa71e503"}
Oct 30 00:31:17.780914 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:17.780881    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpvjg"
Oct 30 00:31:17.781098 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:17.781016    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpvjg" podUID="b3705749-d093-4b49-878d-e4616e81a450"
Oct 30 00:31:17.781426 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:17.781405    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jldmt"
Oct 30 00:31:17.781545 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:17.781521    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jldmt" podUID="e552ba30-5b09-48a7-bffb-6a65a094c2bc"
Oct 30 00:31:17.781676 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:17.781659    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:17.781811 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:17.781747    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f27vk" podUID="3b0df054-0749-4417-bb8b-b12b3353d25e"
Oct 30 00:31:17.833536 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:17.833467    2548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6wzd9"
Oct 30 00:31:17.834517 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:17.834328    2548 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6wzd9"
Oct 30 00:31:17.865832 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:17.864911    2548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6wzd9" podStartSLOduration=5.034371399 podStartE2EDuration="8.86489872s" totalImagesPullingTime="3.830527321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:31:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:17.059673501 +0000 UTC m=+7.686489327" watchObservedRunningTime="2026-10-30 00:31:17.86489872 +0000 UTC m=+8.491714540"
Oct 30 00:31:18.029292 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:18.029202    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz" event={"ID":"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc","Type":"ContainerStarted","Data":"b9f9f9e5a2f6288990debf9a78f51e0b374e27529394ae0de3c3756e01afaf22"}
Oct 30 00:31:18.033115 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:18.033087    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9hslh" event={"ID":"30e43c48-92a7-4c68-887c-9aaf7d6d1e1b","Type":"ContainerStarted","Data":"fb6f3238fbaf5199c98cf7783a1ee8ac008a3d2b3af5a517a2add2fe482d11ab"}
Oct 30 00:31:18.063982 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:18.063917    2548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9hslh" podStartSLOduration=4.284202432 podStartE2EDuration="9.063898293s" totalImagesPullingTime="4.779695861s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:31:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:18.063424556 +0000 UTC m=+8.690240383" watchObservedRunningTime="2026-10-30 00:31:18.063898293 +0000 UTC m=+8.690714121"
Oct 30 00:31:19.036919 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:19.036888    2548 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 00:31:19.493474 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:19.492754    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e552ba30-5b09-48a7-bffb-6a65a094c2bc-metrics-certs\") pod \"network-metrics-daemon-jldmt\" (UID: \"e552ba30-5b09-48a7-bffb-6a65a094c2bc\") " pod="openshift-multus/network-metrics-daemon-jldmt"
Oct 30 00:31:19.493474 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:19.492841    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b0df054-0749-4417-bb8b-b12b3353d25e-original-pull-secret\") pod \"global-pull-secret-syncer-f27vk\" (UID: \"3b0df054-0749-4417-bb8b-b12b3353d25e\") " pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:19.493474 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:19.492968    2548 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:31:19.493474 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:19.493026    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b0df054-0749-4417-bb8b-b12b3353d25e-original-pull-secret podName:3b0df054-0749-4417-bb8b-b12b3353d25e nodeName:}" failed. No retries permitted until 2026-10-30 00:31:27.493010494 +0000 UTC m=+18.119826311 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b0df054-0749-4417-bb8b-b12b3353d25e-original-pull-secret") pod "global-pull-secret-syncer-f27vk" (UID: "3b0df054-0749-4417-bb8b-b12b3353d25e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:31:19.493474 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:19.493387    2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:31:19.493474 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:19.493436    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e552ba30-5b09-48a7-bffb-6a65a094c2bc-metrics-certs podName:e552ba30-5b09-48a7-bffb-6a65a094c2bc nodeName:}" failed. No retries permitted until 2026-10-30 00:31:27.493422215 +0000 UTC m=+18.120238034 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e552ba30-5b09-48a7-bffb-6a65a094c2bc-metrics-certs") pod "network-metrics-daemon-jldmt" (UID: "e552ba30-5b09-48a7-bffb-6a65a094c2bc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:31:19.499458 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:19.499409    2548 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 00:31:19.593491 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:19.593400    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xvts\" (UniqueName: \"kubernetes.io/projected/b3705749-d093-4b49-878d-e4616e81a450-kube-api-access-7xvts\") pod \"network-check-target-wpvjg\" (UID: \"b3705749-d093-4b49-878d-e4616e81a450\") " pod="openshift-network-diagnostics/network-check-target-wpvjg"
Oct 30 00:31:19.593652 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:19.593570    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:31:19.593652 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:19.593585    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:31:19.593652 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:19.593594    2548 projected.go:196] Error preparing data for projected volume kube-api-access-7xvts for pod openshift-network-diagnostics/network-check-target-wpvjg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:31:19.593652 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:19.593639    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3705749-d093-4b49-878d-e4616e81a450-kube-api-access-7xvts podName:b3705749-d093-4b49-878d-e4616e81a450 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:27.593625135 +0000 UTC m=+18.220440953 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7xvts" (UniqueName: "kubernetes.io/projected/b3705749-d093-4b49-878d-e4616e81a450-kube-api-access-7xvts") pod "network-check-target-wpvjg" (UID: "b3705749-d093-4b49-878d-e4616e81a450") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:31:19.781723 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:19.781116    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jldmt"
Oct 30 00:31:19.781723 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:19.781197    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:19.781723 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:19.781224    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpvjg"
Oct 30 00:31:19.781723 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:19.781314    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f27vk" podUID="3b0df054-0749-4417-bb8b-b12b3353d25e"
Oct 30 00:31:19.781723 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:19.781423    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpvjg" podUID="b3705749-d093-4b49-878d-e4616e81a450"
Oct 30 00:31:19.781723 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:19.781511    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jldmt" podUID="e552ba30-5b09-48a7-bffb-6a65a094c2bc"
Oct 30 00:31:19.793250 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:19.793146    2548 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T00:31:19.499425696Z","UUID":"10aaa824-d471-4dd7-9b76-598fb783945a","Handler":null,"Name":"","Endpoint":""}
Oct 30 00:31:19.796595 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:19.795283    2548 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 00:31:19.796595 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:19.795321    2548 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 00:31:20.054984 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:20.054938    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz" event={"ID":"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc","Type":"ContainerStarted","Data":"724a789450de021b500667e8d6e07b5372462a76c63aa0230dbb799e6f402136"}
Oct 30 00:31:20.057314 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:20.057287    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r58sp" event={"ID":"3b734765-8e7f-4d8e-a74b-bd179adf33ee","Type":"ContainerStarted","Data":"cbe8535aa756dd72f472f3f8073dfb8f13cb2fd417c0e7069fd1db4d09523c61"}
Oct 30 00:31:20.092541 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:20.092241    2548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-r58sp" podStartSLOduration=3.32812308 podStartE2EDuration="11.092224096s" totalImagesPullingTime="7.764101016s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:31:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:20.09192726 +0000 UTC m=+10.718743086" watchObservedRunningTime="2026-10-30 00:31:20.092224096 +0000 UTC m=+10.719039923"
Oct 30 00:31:21.063445 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:21.063355    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz" event={"ID":"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc","Type":"ContainerStarted","Data":"7240b6567822a52c8a0bd5e9bb3e46575a3fc42e90be2f128365c2865b69a9fd"}
Oct 30 00:31:21.065845 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:21.065810    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fqtzs" event={"ID":"05a614f5-3112-4216-b894-68bf3945458c","Type":"ContainerStarted","Data":"66d60784eb56861cab1b69153c46223d97fde44751fc75d2ca879e488700499e"}
Oct 30 00:31:21.068082 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:21.068046    2548 generic.go:356] "Generic (PLEG): container finished" podID="ef9a0a39-051f-45a1-acc9-354a23e5b00b" containerID="13dc4a5cea096cfe80d09b3d12f5bbe3bb607811d3eecd9c65c5dd4fe5f42a78" exitCode=0
Oct 30 00:31:21.068256 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:21.068210    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-csc84" event={"ID":"ef9a0a39-051f-45a1-acc9-354a23e5b00b","Type":"ContainerDied","Data":"13dc4a5cea096cfe80d09b3d12f5bbe3bb607811d3eecd9c65c5dd4fe5f42a78"}
Oct 30 00:31:21.098022 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:21.097962    2548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz" podStartSLOduration=3.930019128 podStartE2EDuration="12.097943533s" totalImagesPullingTime="8.167924405s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:31:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:21.097381554 +0000 UTC m=+11.724197386" watchObservedRunningTime="2026-10-30 00:31:21.097943533 +0000 UTC m=+11.724759361"
Oct 30 00:31:21.129874 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:21.129800    2548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-fqtzs" podStartSLOduration=4.188612967 podStartE2EDuration="12.129760887s" totalImagesPullingTime="7.94114792s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:31:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:21.12952277 +0000 UTC m=+11.756338600" watchObservedRunningTime="2026-10-30 00:31:21.129760887 +0000 UTC m=+11.756576713"
Oct 30 00:31:21.779961 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:21.779932    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jldmt"
Oct 30 00:31:21.780142 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:21.779979    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:21.780142 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:21.780070    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jldmt" podUID="e552ba30-5b09-48a7-bffb-6a65a094c2bc"
Oct 30 00:31:21.780505 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:21.780475    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f27vk" podUID="3b0df054-0749-4417-bb8b-b12b3353d25e"
Oct 30 00:31:21.780605 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:21.780506    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpvjg"
Oct 30 00:31:21.780661 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:21.780637    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpvjg" podUID="b3705749-d093-4b49-878d-e4616e81a450"
Oct 30 00:31:22.072295 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:22.072203    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s72l8" event={"ID":"4a99683d-3f83-465d-9152-2f422641e087","Type":"ContainerStarted","Data":"cb88929338142a38c374bc553af97b8b89412b9efd8891ae8f882c48295bdbe8"}
Oct 30 00:31:22.103701 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:22.103643    2548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-s72l8" podStartSLOduration=5.496052877 podStartE2EDuration="13.103627444s" totalImagesPullingTime="7.607574567s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:31:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:22.10321127 +0000 UTC m=+12.730027097" watchObservedRunningTime="2026-10-30 00:31:22.103627444 +0000 UTC m=+12.730443269"
Oct 30 00:31:23.075872 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:23.075817    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lnnlc" event={"ID":"5687019d-945a-4307-8269-5c0ae6f377ec","Type":"ContainerStarted","Data":"6ae16149ee6c8920adf83116894cd85bf714c539b804b65819db660cdd574840"}
Oct 30 00:31:23.078271 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:23.078237    2548 generic.go:356] "Generic (PLEG): container finished" podID="ef9a0a39-051f-45a1-acc9-354a23e5b00b" containerID="eb660c3e5e3926db0ed2cca4dbce6808d91745769650428aa70cf4b5b4654c8d" exitCode=0
Oct 30 00:31:23.078417 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:23.078341    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-csc84" event={"ID":"ef9a0a39-051f-45a1-acc9-354a23e5b00b","Type":"ContainerDied","Data":"eb660c3e5e3926db0ed2cca4dbce6808d91745769650428aa70cf4b5b4654c8d"}
Oct 30 00:31:23.109110 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:23.109063    2548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lnnlc" podStartSLOduration=4.005262937 podStartE2EDuration="14.109043481s" totalImagesPullingTime="10.103780544s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:31:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:23.108903355 +0000 UTC m=+13.735719182" watchObservedRunningTime="2026-10-30 00:31:23.109043481 +0000 UTC m=+13.735859304"
Oct 30 00:31:23.780528 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:23.779676    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:23.780528 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:23.779711    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jldmt"
Oct 30 00:31:23.780528 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:23.779725    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpvjg"
Oct 30 00:31:23.780528 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:23.779856    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f27vk" podUID="3b0df054-0749-4417-bb8b-b12b3353d25e"
Oct 30 00:31:23.780528 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:23.780391    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jldmt" podUID="e552ba30-5b09-48a7-bffb-6a65a094c2bc"
Oct 30 00:31:23.780528 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:23.780489    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpvjg" podUID="b3705749-d093-4b49-878d-e4616e81a450"
Oct 30 00:31:24.084753 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:24.084646    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-csc84" event={"ID":"ef9a0a39-051f-45a1-acc9-354a23e5b00b","Type":"ContainerStarted","Data":"a115a4e49fdd94fd1c4243efca7324e804b57a9d62324c2563c678d18950a5c7"}
Oct 30 00:31:25.090293 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:25.090066    2548 generic.go:356] "Generic (PLEG): container finished" podID="ef9a0a39-051f-45a1-acc9-354a23e5b00b" containerID="a115a4e49fdd94fd1c4243efca7324e804b57a9d62324c2563c678d18950a5c7" exitCode=0
Oct 30 00:31:25.090819 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:25.090321    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-csc84" event={"ID":"ef9a0a39-051f-45a1-acc9-354a23e5b00b","Type":"ContainerDied","Data":"a115a4e49fdd94fd1c4243efca7324e804b57a9d62324c2563c678d18950a5c7"}
Oct 30 00:31:25.779938 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:25.779904    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpvjg"
Oct 30 00:31:25.780135 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:25.780044    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpvjg" podUID="b3705749-d093-4b49-878d-e4616e81a450"
Oct 30 00:31:25.780477 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:25.780458    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jldmt"
Oct 30 00:31:25.780602 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:25.780582    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jldmt" podUID="e552ba30-5b09-48a7-bffb-6a65a094c2bc"
Oct 30 00:31:25.780741 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:25.780728    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:25.780851 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:25.780832    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f27vk" podUID="3b0df054-0749-4417-bb8b-b12b3353d25e"
Oct 30 00:31:27.098718 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:27.098461    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp" event={"ID":"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835","Type":"ContainerStarted","Data":"709525bbabc8153b3c557a47e66a8edef0ef602b66cd39aac385faa7a8613efe"}
Oct 30 00:31:27.098718 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:27.098704    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp" event={"ID":"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835","Type":"ContainerStarted","Data":"882e3d4dde7aaa1dbafa0e36abc32a0d9698557a5429f442fe4bff43377640de"}
Oct 30 00:31:27.099218 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:27.098739    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp" event={"ID":"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835","Type":"ContainerStarted","Data":"ad6e6b108a82526417a7a4ca0068f4de93946f274057f231c42e32b21ef220b7"}
Oct 30 00:31:27.099218 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:27.098808    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp" event={"ID":"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835","Type":"ContainerStarted","Data":"38d243c052c82f55c8d729ae74d63c49f2057ef67e26d59e7d2f96886c244d4f"}
Oct 30 00:31:27.099218 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:27.098855    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp" event={"ID":"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835","Type":"ContainerStarted","Data":"a4c0c43769b55b2827f655159b902e3f23af32df128d3a524e89796ff45ca16a"}
Oct 30 00:31:27.099218 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:27.098902    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp" event={"ID":"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835","Type":"ContainerStarted","Data":"2952d5b0a0df42854fddbbac53e52da161f130f0bfcb83ba2e69af2626f068be"}
Oct 30 00:31:27.104009 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:27.103980    2548 generic.go:356] "Generic (PLEG): container finished" podID="ef9a0a39-051f-45a1-acc9-354a23e5b00b" containerID="2740e9825f42fce105efafe77efdce4b6a01bcde5ed71d79fa66c9e683f1120b" exitCode=0
Oct 30 00:31:27.104131 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:27.104029    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-csc84" event={"ID":"ef9a0a39-051f-45a1-acc9-354a23e5b00b","Type":"ContainerDied","Data":"2740e9825f42fce105efafe77efdce4b6a01bcde5ed71d79fa66c9e683f1120b"}
Oct 30 00:31:27.553909 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:27.553850    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b0df054-0749-4417-bb8b-b12b3353d25e-original-pull-secret\") pod \"global-pull-secret-syncer-f27vk\" (UID: \"3b0df054-0749-4417-bb8b-b12b3353d25e\") " pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:27.554099 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:27.553947    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e552ba30-5b09-48a7-bffb-6a65a094c2bc-metrics-certs\") pod \"network-metrics-daemon-jldmt\" (UID: \"e552ba30-5b09-48a7-bffb-6a65a094c2bc\") " pod="openshift-multus/network-metrics-daemon-jldmt"
Oct 30 00:31:27.554099 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:27.554020    2548 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 00:31:27.554099 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:27.554033    2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:31:27.554099 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:27.554086    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e552ba30-5b09-48a7-bffb-6a65a094c2bc-metrics-certs podName:e552ba30-5b09-48a7-bffb-6a65a094c2bc nodeName:}" failed. No retries permitted until 2026-10-30 00:31:43.554073157 +0000 UTC m=+34.180888961 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e552ba30-5b09-48a7-bffb-6a65a094c2bc-metrics-certs") pod "network-metrics-daemon-jldmt" (UID: "e552ba30-5b09-48a7-bffb-6a65a094c2bc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 00:31:27.554099 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:27.554099    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b0df054-0749-4417-bb8b-b12b3353d25e-original-pull-secret podName:3b0df054-0749-4417-bb8b-b12b3353d25e nodeName:}" failed. No retries permitted until 2026-10-30 00:31:43.554093054 +0000 UTC m=+34.180908858 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b0df054-0749-4417-bb8b-b12b3353d25e-original-pull-secret") pod "global-pull-secret-syncer-f27vk" (UID: "3b0df054-0749-4417-bb8b-b12b3353d25e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 00:31:27.654980 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:27.654944    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xvts\" (UniqueName: \"kubernetes.io/projected/b3705749-d093-4b49-878d-e4616e81a450-kube-api-access-7xvts\") pod \"network-check-target-wpvjg\" (UID: \"b3705749-d093-4b49-878d-e4616e81a450\") " pod="openshift-network-diagnostics/network-check-target-wpvjg"
Oct 30 00:31:27.655174 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:27.655074    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 00:31:27.655174 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:27.655091    2548 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 00:31:27.655174 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:27.655100    2548 projected.go:196] Error preparing data for projected volume kube-api-access-7xvts for pod openshift-network-diagnostics/network-check-target-wpvjg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:31:27.655174 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:27.655154    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3705749-d093-4b49-878d-e4616e81a450-kube-api-access-7xvts podName:b3705749-d093-4b49-878d-e4616e81a450 nodeName:}" failed. No retries permitted until 2026-10-30 00:31:43.655138299 +0000 UTC m=+34.281954103 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7xvts" (UniqueName: "kubernetes.io/projected/b3705749-d093-4b49-878d-e4616e81a450-kube-api-access-7xvts") pod "network-check-target-wpvjg" (UID: "b3705749-d093-4b49-878d-e4616e81a450") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 00:31:27.780121 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:27.780032    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jldmt"
Oct 30 00:31:27.780121 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:27.780089    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpvjg"
Oct 30 00:31:27.780121 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:27.780095    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:27.780321 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:27.780192    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jldmt" podUID="e552ba30-5b09-48a7-bffb-6a65a094c2bc"
Oct 30 00:31:27.780321 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:27.780304    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpvjg" podUID="b3705749-d093-4b49-878d-e4616e81a450"
Oct 30 00:31:27.780399 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:27.780382    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f27vk" podUID="3b0df054-0749-4417-bb8b-b12b3353d25e"
Oct 30 00:31:28.109248 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:28.109159    2548 generic.go:356] "Generic (PLEG): container finished" podID="ef9a0a39-051f-45a1-acc9-354a23e5b00b" containerID="6b8776286e3ae617ffd413c0780fb003bbf21e2d57d4914514298b6f408ebc32" exitCode=0
Oct 30 00:31:28.109248 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:28.109206    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-csc84" event={"ID":"ef9a0a39-051f-45a1-acc9-354a23e5b00b","Type":"ContainerDied","Data":"6b8776286e3ae617ffd413c0780fb003bbf21e2d57d4914514298b6f408ebc32"}
Oct 30 00:31:29.115249 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:29.115211    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp" event={"ID":"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835","Type":"ContainerStarted","Data":"8d5b12b31b43bfe5847bf5815eec56414e94b7572429ca6c7d9f5df2030a569c"}
Oct 30 00:31:29.118396 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:29.118367    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-csc84" event={"ID":"ef9a0a39-051f-45a1-acc9-354a23e5b00b","Type":"ContainerStarted","Data":"58695a8c4f4a500174858d97808f98e9a83b4667e6f1c556597a826694c11a4f"}
Oct 30 00:31:29.179196 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:29.179097    2548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-csc84" podStartSLOduration=8.789589981 podStartE2EDuration="19.179083071s" totalImagesPullingTime="10.38949309s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:31:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:29.168972944 +0000 UTC m=+19.795788767" watchObservedRunningTime="2026-10-30 00:31:29.179083071 +0000 UTC m=+19.805898897"
Oct 30 00:31:29.780478 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:29.780444    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:29.780629 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:29.780542    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f27vk" podUID="3b0df054-0749-4417-bb8b-b12b3353d25e"
Oct 30 00:31:29.780629 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:29.780592    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jldmt"
Oct 30 00:31:29.780629 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:29.780602    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpvjg"
Oct 30 00:31:29.780719 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:29.780679    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jldmt" podUID="e552ba30-5b09-48a7-bffb-6a65a094c2bc"
Oct 30 00:31:29.780773 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:29.780732    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpvjg" podUID="b3705749-d093-4b49-878d-e4616e81a450"
Oct 30 00:31:31.126929 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:31.126893    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp" event={"ID":"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835","Type":"ContainerStarted","Data":"e482f87b4dd2301058c681ae9f0f8f6d8e9bcc7680c9535be1ee55ebf203e218"}
Oct 30 00:31:31.127542 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:31.127176    2548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:31.127542 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:31.127262    2548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:31.127542 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:31.127316    2548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:31.144485 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:31.144346    2548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:31.144611 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:31.144568    2548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:31.202273 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:31.202217    2548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp" podStartSLOduration=7.804944643 podStartE2EDuration="22.202204061s" totalImagesPullingTime="14.397259418s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 00:31:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:31.201665119 +0000 UTC m=+21.828480944" watchObservedRunningTime="2026-10-30 00:31:31.202204061 +0000 UTC m=+21.829019919"
Oct 30 00:31:31.508678 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:31.508640    2548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6wzd9"
Oct 30 00:31:31.508926 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:31.508762    2548 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 00:31:31.509382 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:31.509366    2548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6wzd9"
Oct 30 00:31:31.780324 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:31.780239    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpvjg"
Oct 30 00:31:31.780458 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:31.780250    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:31.780458 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:31.780362    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jldmt"
Oct 30 00:31:31.780458 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:31.780362    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpvjg" podUID="b3705749-d093-4b49-878d-e4616e81a450"
Oct 30 00:31:31.780458 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:31.780440    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f27vk" podUID="3b0df054-0749-4417-bb8b-b12b3353d25e"
Oct 30 00:31:31.780677 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:31.780527    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jldmt" podUID="e552ba30-5b09-48a7-bffb-6a65a094c2bc"
Oct 30 00:31:33.070008 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:33.069915    2548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f27vk"]
Oct 30 00:31:33.070545 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:33.070038    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:33.070545 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:33.070161    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f27vk" podUID="3b0df054-0749-4417-bb8b-b12b3353d25e"
Oct 30 00:31:33.075250 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:33.075221    2548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wpvjg"]
Oct 30 00:31:33.075353 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:33.075323    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpvjg"
Oct 30 00:31:33.075446 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:33.075424    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpvjg" podUID="b3705749-d093-4b49-878d-e4616e81a450"
Oct 30 00:31:33.079219 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:33.079194    2548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jldmt"]
Oct 30 00:31:33.079338 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:33.079324    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jldmt"
Oct 30 00:31:33.079446 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:33.079428    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jldmt" podUID="e552ba30-5b09-48a7-bffb-6a65a094c2bc"
Oct 30 00:31:34.780185 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:34.780149    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:34.780185 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:34.780185    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpvjg"
Oct 30 00:31:34.780647 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:34.780257    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f27vk" podUID="3b0df054-0749-4417-bb8b-b12b3353d25e"
Oct 30 00:31:34.780647 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:34.780305    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jldmt"
Oct 30 00:31:34.780647 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:34.780388    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jldmt" podUID="e552ba30-5b09-48a7-bffb-6a65a094c2bc"
Oct 30 00:31:34.780647 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:34.780444    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpvjg" podUID="b3705749-d093-4b49-878d-e4616e81a450"
Oct 30 00:31:36.780208 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:36.780166    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpvjg"
Oct 30 00:31:36.780208 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:36.780200    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:36.780799 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:36.780286    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jldmt"
Oct 30 00:31:36.780799 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:36.780289    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpvjg" podUID="b3705749-d093-4b49-878d-e4616e81a450"
Oct 30 00:31:36.780799 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:36.780399    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f27vk" podUID="3b0df054-0749-4417-bb8b-b12b3353d25e"
Oct 30 00:31:36.780799 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:36.780462    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jldmt" podUID="e552ba30-5b09-48a7-bffb-6a65a094c2bc"
Oct 30 00:31:38.779673 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:38.779643    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:38.780085 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:38.779643    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpvjg"
Oct 30 00:31:38.780085 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:38.779747    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f27vk" podUID="3b0df054-0749-4417-bb8b-b12b3353d25e"
Oct 30 00:31:38.780085 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:38.779864    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpvjg" podUID="b3705749-d093-4b49-878d-e4616e81a450"
Oct 30 00:31:38.780085 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:38.779877    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jldmt"
Oct 30 00:31:38.780085 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:38.779985    2548 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jldmt" podUID="e552ba30-5b09-48a7-bffb-6a65a094c2bc"
Oct 30 00:31:39.672943 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.672733    2548 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-56.ec2.internal" event="NodeReady"
Oct 30 00:31:39.673128 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.673055    2548 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 00:31:39.731698 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.731665    2548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jb5sp"]
Oct 30 00:31:39.735993 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.735968    2548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9ml2g"]
Oct 30 00:31:39.736130 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.736085    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jb5sp"
Oct 30 00:31:39.737976 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.737952    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9ml2g"
Oct 30 00:31:39.753299 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.753270    2548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jb5sp"]
Oct 30 00:31:39.755440 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.755408    2548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 00:31:39.755588 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.755467    2548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 00:31:39.755588 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.755474    2548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 00:31:39.755588 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.755496    2548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-nrfgz\""
Oct 30 00:31:39.755588 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.755521    2548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 00:31:39.755588 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.755554    2548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fch7k\""
Oct 30 00:31:39.755794 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.755757    2548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 00:31:39.764812 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.764753    2548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jbmfs"]
Oct 30 00:31:39.768603 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.768569    2548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9ml2g"]
Oct 30 00:31:39.768769 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.768738    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:39.782969 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.782935    2548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jbmfs"]
Oct 30 00:31:39.834276 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.834244    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a20a733f-4947-4a14-9db0-a5be23cb9ccb-cert\") pod \"ingress-canary-jb5sp\" (UID: \"a20a733f-4947-4a14-9db0-a5be23cb9ccb\") " pod="openshift-ingress-canary/ingress-canary-jb5sp"
Oct 30 00:31:39.834455 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.834294    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:39.834455 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.834369    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-metrics-tls\") pod \"dns-default-9ml2g\" (UID: \"b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f\") " pod="openshift-dns/dns-default-9ml2g"
Oct 30 00:31:39.834455 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.834394    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-tmp-dir\") pod \"dns-default-9ml2g\" (UID: \"b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f\") " pod="openshift-dns/dns-default-9ml2g"
Oct 30 00:31:39.834455 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.834437    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:39.834590 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.834457    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdxj7\" (UniqueName: \"kubernetes.io/projected/a20a733f-4947-4a14-9db0-a5be23cb9ccb-kube-api-access-fdxj7\") pod \"ingress-canary-jb5sp\" (UID: \"a20a733f-4947-4a14-9db0-a5be23cb9ccb\") " pod="openshift-ingress-canary/ingress-canary-jb5sp"
Oct 30 00:31:39.834590 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.834508    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tntlm\" (UniqueName: \"kubernetes.io/projected/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-kube-api-access-tntlm\") pod \"dns-default-9ml2g\" (UID: \"b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f\") " pod="openshift-dns/dns-default-9ml2g"
Oct 30 00:31:39.834652 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.834588    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-config-volume\") pod \"dns-default-9ml2g\" (UID: \"b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f\") " pod="openshift-dns/dns-default-9ml2g"
Oct 30 00:31:39.834652 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.834608    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-data-volume\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:39.834709 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.834653    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5mw6\" (UniqueName: \"kubernetes.io/projected/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-kube-api-access-g5mw6\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:39.834709 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.834674    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-crio-socket\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:39.854366 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.854329    2548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 00:31:39.854366 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.854338    2548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 00:31:39.854899 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.854882    2548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 00:31:39.854977 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.854898    2548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 00:31:39.855210 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.855194    2548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ngd4v\""
Oct 30 00:31:39.935491 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.935396    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-config-volume\") pod \"dns-default-9ml2g\" (UID: \"b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f\") " pod="openshift-dns/dns-default-9ml2g"
Oct 30 00:31:39.935491 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.935431    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-data-volume\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:39.935491 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.935452    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5mw6\" (UniqueName: \"kubernetes.io/projected/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-kube-api-access-g5mw6\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:39.935773 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.935631    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-crio-socket\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:39.935773 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.935728    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a20a733f-4947-4a14-9db0-a5be23cb9ccb-cert\") pod \"ingress-canary-jb5sp\" (UID: \"a20a733f-4947-4a14-9db0-a5be23cb9ccb\") " pod="openshift-ingress-canary/ingress-canary-jb5sp"
Oct 30 00:31:39.935903 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.935773    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:39.935903 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.935841    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-metrics-tls\") pod \"dns-default-9ml2g\" (UID: \"b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f\") " pod="openshift-dns/dns-default-9ml2g"
Oct 30 00:31:39.935903 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.935858    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-data-volume\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:39.935903 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.935869    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-tmp-dir\") pod \"dns-default-9ml2g\" (UID: \"b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f\") " pod="openshift-dns/dns-default-9ml2g"
Oct 30 00:31:39.935903 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.935883    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-crio-socket\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:39.935903 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.935897    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:39.936160 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:39.935959    2548 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:31:39.936160 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:39.935962    2548 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 00:31:39.936160 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.935993    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdxj7\" (UniqueName: \"kubernetes.io/projected/a20a733f-4947-4a14-9db0-a5be23cb9ccb-kube-api-access-fdxj7\") pod \"ingress-canary-jb5sp\" (UID: \"a20a733f-4947-4a14-9db0-a5be23cb9ccb\") " pod="openshift-ingress-canary/ingress-canary-jb5sp"
Oct 30 00:31:39.936160 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:39.936023    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-insights-runtime-extractor-tls podName:626ec6c0-d616-40ae-bf8b-4e8a927c31ac nodeName:}" failed. No retries permitted until 2026-10-30 00:31:40.436003005 +0000 UTC m=+31.062818824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jbmfs" (UID: "626ec6c0-d616-40ae-bf8b-4e8a927c31ac") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:31:39.936160 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:39.936113    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-kube-rbac-proxy-cm podName:626ec6c0-d616-40ae-bf8b-4e8a927c31ac nodeName:}" failed. No retries permitted until 2026-10-30 00:31:40.436096671 +0000 UTC m=+31.062912482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jbmfs" (UID: "626ec6c0-d616-40ae-bf8b-4e8a927c31ac") : configmap "kube-rbac-proxy" not found
Oct 30 00:31:39.936160 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.936154    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tntlm\" (UniqueName: \"kubernetes.io/projected/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-kube-api-access-tntlm\") pod \"dns-default-9ml2g\" (UID: \"b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f\") " pod="openshift-dns/dns-default-9ml2g"
Oct 30 00:31:39.936414 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.936172    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-tmp-dir\") pod \"dns-default-9ml2g\" (UID: \"b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f\") " pod="openshift-dns/dns-default-9ml2g"
Oct 30 00:31:39.936414 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.936184    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-config-volume\") pod \"dns-default-9ml2g\" (UID: \"b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f\") " pod="openshift-dns/dns-default-9ml2g"
Oct 30 00:31:39.936414 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:39.936313    2548 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:31:39.936414 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:39.936380    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-metrics-tls podName:b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f nodeName:}" failed. No retries permitted until 2026-10-30 00:31:40.436363589 +0000 UTC m=+31.063179397 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-metrics-tls") pod "dns-default-9ml2g" (UID: "b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f") : secret "dns-default-metrics-tls" not found
Oct 30 00:31:39.940959 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.940931    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a20a733f-4947-4a14-9db0-a5be23cb9ccb-cert\") pod \"ingress-canary-jb5sp\" (UID: \"a20a733f-4947-4a14-9db0-a5be23cb9ccb\") " pod="openshift-ingress-canary/ingress-canary-jb5sp"
Oct 30 00:31:39.958965 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.958926    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tntlm\" (UniqueName: \"kubernetes.io/projected/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-kube-api-access-tntlm\") pod \"dns-default-9ml2g\" (UID: \"b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f\") " pod="openshift-dns/dns-default-9ml2g"
Oct 30 00:31:39.959112 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.958926    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5mw6\" (UniqueName: \"kubernetes.io/projected/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-kube-api-access-g5mw6\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:39.959112 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:39.959024    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdxj7\" (UniqueName: \"kubernetes.io/projected/a20a733f-4947-4a14-9db0-a5be23cb9ccb-kube-api-access-fdxj7\") pod \"ingress-canary-jb5sp\" (UID: \"a20a733f-4947-4a14-9db0-a5be23cb9ccb\") " pod="openshift-ingress-canary/ingress-canary-jb5sp"
Oct 30 00:31:40.047889 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:40.047852    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jb5sp"
Oct 30 00:31:40.350982 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:40.350895    2548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jb5sp"]
Oct 30 00:31:40.356070 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:40.356038    2548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda20a733f_4947_4a14_9db0_a5be23cb9ccb.slice/crio-b78879c21b93d6a353f9ed0d9d651ec4036b9443832f9c1456ee3d7e0cad7402 WatchSource:0}: Error finding container b78879c21b93d6a353f9ed0d9d651ec4036b9443832f9c1456ee3d7e0cad7402: Status 404 returned error can't find the container with id b78879c21b93d6a353f9ed0d9d651ec4036b9443832f9c1456ee3d7e0cad7402
Oct 30 00:31:40.439852 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:40.439769    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:40.440038 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:40.439872    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-metrics-tls\") pod \"dns-default-9ml2g\" (UID: \"b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f\") " pod="openshift-dns/dns-default-9ml2g"
Oct 30 00:31:40.440038 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:40.439904    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:40.440038 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:40.439957    2548 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:31:40.440038 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:40.439992    2548 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:31:40.440038 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:40.439996    2548 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 00:31:40.440224 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:40.440044    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-insights-runtime-extractor-tls podName:626ec6c0-d616-40ae-bf8b-4e8a927c31ac nodeName:}" failed. No retries permitted until 2026-10-30 00:31:41.440022587 +0000 UTC m=+32.066838395 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jbmfs" (UID: "626ec6c0-d616-40ae-bf8b-4e8a927c31ac") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:31:40.440224 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:40.440060    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-metrics-tls podName:b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f nodeName:}" failed. No retries permitted until 2026-10-30 00:31:41.44005301 +0000 UTC m=+32.066868813 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-metrics-tls") pod "dns-default-9ml2g" (UID: "b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f") : secret "dns-default-metrics-tls" not found
Oct 30 00:31:40.440224 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:40.440069    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-kube-rbac-proxy-cm podName:626ec6c0-d616-40ae-bf8b-4e8a927c31ac nodeName:}" failed. No retries permitted until 2026-10-30 00:31:41.440064313 +0000 UTC m=+32.066880117 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jbmfs" (UID: "626ec6c0-d616-40ae-bf8b-4e8a927c31ac") : configmap "kube-rbac-proxy" not found
Oct 30 00:31:40.557232 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:40.557195    2548 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-mln7q"]
Oct 30 00:31:40.559479 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:40.559459    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-mln7q"
Oct 30 00:31:40.573258 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:40.573233    2548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 00:31:40.580198 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:40.580162    2548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-qfggf\""
Oct 30 00:31:40.580351 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:40.580319    2548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 00:31:40.580351 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:40.580344    2548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 00:31:40.580466 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:40.580352    2548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 00:31:40.581410 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:40.581392    2548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-mln7q"]
Oct 30 00:31:40.641960 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:40.641932    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8ff169ff-dc1a-4344-b19b-adaabe98f20d-signing-key\") pod \"service-ca-5fffc97bdf-mln7q\" (UID: \"8ff169ff-dc1a-4344-b19b-adaabe98f20d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mln7q"
Oct 30 00:31:40.642154 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:40.641977    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4nwk\" (UniqueName: \"kubernetes.io/projected/8ff169ff-dc1a-4344-b19b-adaabe98f20d-kube-api-access-c4nwk\") pod \"service-ca-5fffc97bdf-mln7q\" (UID: \"8ff169ff-dc1a-4344-b19b-adaabe98f20d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mln7q"
Oct 30 00:31:40.642154 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:40.641998    2548 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8ff169ff-dc1a-4344-b19b-adaabe98f20d-signing-cabundle\") pod \"service-ca-5fffc97bdf-mln7q\" (UID: \"8ff169ff-dc1a-4344-b19b-adaabe98f20d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mln7q"
Oct 30 00:31:40.743050 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:40.743008    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8ff169ff-dc1a-4344-b19b-adaabe98f20d-signing-key\") pod \"service-ca-5fffc97bdf-mln7q\" (UID: \"8ff169ff-dc1a-4344-b19b-adaabe98f20d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mln7q"
Oct 30 00:31:40.743248 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:40.743074    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4nwk\" (UniqueName: \"kubernetes.io/projected/8ff169ff-dc1a-4344-b19b-adaabe98f20d-kube-api-access-c4nwk\") pod \"service-ca-5fffc97bdf-mln7q\" (UID: \"8ff169ff-dc1a-4344-b19b-adaabe98f20d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mln7q"
Oct 30 00:31:40.743248 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:40.743106    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8ff169ff-dc1a-4344-b19b-adaabe98f20d-signing-cabundle\") pod \"service-ca-5fffc97bdf-mln7q\" (UID: \"8ff169ff-dc1a-4344-b19b-adaabe98f20d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mln7q"
Oct 30 00:31:40.744069 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:40.744033    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8ff169ff-dc1a-4344-b19b-adaabe98f20d-signing-cabundle\") pod \"service-ca-5fffc97bdf-mln7q\" (UID: \"8ff169ff-dc1a-4344-b19b-adaabe98f20d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mln7q"
Oct 30 00:31:40.746438 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:40.746405    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8ff169ff-dc1a-4344-b19b-adaabe98f20d-signing-key\") pod \"service-ca-5fffc97bdf-mln7q\" (UID: \"8ff169ff-dc1a-4344-b19b-adaabe98f20d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mln7q"
Oct 30 00:31:40.764010 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:40.763974    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4nwk\" (UniqueName: \"kubernetes.io/projected/8ff169ff-dc1a-4344-b19b-adaabe98f20d-kube-api-access-c4nwk\") pod \"service-ca-5fffc97bdf-mln7q\" (UID: \"8ff169ff-dc1a-4344-b19b-adaabe98f20d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mln7q"
Oct 30 00:31:40.780023 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:40.779993    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:40.780023 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:40.780011    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jldmt"
Oct 30 00:31:40.780023 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:40.780033    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpvjg"
Oct 30 00:31:40.792396 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:40.792375    2548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 00:31:40.792754 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:40.792402    2548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-xrsjq\""
Oct 30 00:31:40.793035 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:40.793015    2548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6gpzw\""
Oct 30 00:31:40.793035 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:40.793030    2548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 00:31:40.793162 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:40.793117    2548 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-rw2kd\""
Oct 30 00:31:40.856302 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:40.856274    2548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 00:31:40.856471 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:40.856327    2548 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 00:31:40.869473 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:40.869440    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-mln7q"
Oct 30 00:31:41.152838 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:41.152802    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jb5sp" event={"ID":"a20a733f-4947-4a14-9db0-a5be23cb9ccb","Type":"ContainerStarted","Data":"b78879c21b93d6a353f9ed0d9d651ec4036b9443832f9c1456ee3d7e0cad7402"}
Oct 30 00:31:41.164811 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:41.164750    2548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-mln7q"]
Oct 30 00:31:41.170254 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:41.170180    2548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ff169ff_dc1a_4344_b19b_adaabe98f20d.slice/crio-0eb49afe8f3d27e8f679c06d48434c3f565f094fbfde9e391b0fe3ee49e57e9a WatchSource:0}: Error finding container 0eb49afe8f3d27e8f679c06d48434c3f565f094fbfde9e391b0fe3ee49e57e9a: Status 404 returned error can't find the container with id 0eb49afe8f3d27e8f679c06d48434c3f565f094fbfde9e391b0fe3ee49e57e9a
Oct 30 00:31:41.450570 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:41.450471    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:41.450570 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:41.450520    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-metrics-tls\") pod \"dns-default-9ml2g\" (UID: \"b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f\") " pod="openshift-dns/dns-default-9ml2g"
Oct 30 00:31:41.450817 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:41.450654    2548 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:31:41.450817 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:41.450655    2548 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:31:41.450817 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:41.450682    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:41.450817 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:41.450712    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-metrics-tls podName:b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f nodeName:}" failed. No retries permitted until 2026-10-30 00:31:43.45069327 +0000 UTC m=+34.077509088 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-metrics-tls") pod "dns-default-9ml2g" (UID: "b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f") : secret "dns-default-metrics-tls" not found
Oct 30 00:31:41.450817 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:41.450735    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-insights-runtime-extractor-tls podName:626ec6c0-d616-40ae-bf8b-4e8a927c31ac nodeName:}" failed. No retries permitted until 2026-10-30 00:31:43.450725385 +0000 UTC m=+34.077541203 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jbmfs" (UID: "626ec6c0-d616-40ae-bf8b-4e8a927c31ac") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:31:41.450817 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:41.450744    2548 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 00:31:41.451025 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:41.450841    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-kube-rbac-proxy-cm podName:626ec6c0-d616-40ae-bf8b-4e8a927c31ac nodeName:}" failed. No retries permitted until 2026-10-30 00:31:43.450824124 +0000 UTC m=+34.077639930 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jbmfs" (UID: "626ec6c0-d616-40ae-bf8b-4e8a927c31ac") : configmap "kube-rbac-proxy" not found
Oct 30 00:31:42.156937 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:42.156901    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-mln7q" event={"ID":"8ff169ff-dc1a-4344-b19b-adaabe98f20d","Type":"ContainerStarted","Data":"0eb49afe8f3d27e8f679c06d48434c3f565f094fbfde9e391b0fe3ee49e57e9a"}
Oct 30 00:31:43.160973 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:43.160936    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jb5sp" event={"ID":"a20a733f-4947-4a14-9db0-a5be23cb9ccb","Type":"ContainerStarted","Data":"faf112ff66661a8d7d5624561493d24640518aaded0e96434fcc65f706c4713b"}
Oct 30 00:31:43.196962 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:43.196900    2548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jb5sp" podStartSLOduration=1.8039051050000001 podStartE2EDuration="4.196884989s" totalImagesPullingTime="2.392979884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:31:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:43.195563702 +0000 UTC m=+33.822379529" watchObservedRunningTime="2026-10-30 00:31:43.196884989 +0000 UTC m=+33.823700814"
Oct 30 00:31:43.266146 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:43.266122    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fqtzs_05a614f5-3112-4216-b894-68bf3945458c/dns-node-resolver/0.log"
Oct 30 00:31:43.465164 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:43.465119    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:43.465358 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:43.465265    2548 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 00:31:43.465358 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:43.465269    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:43.465358 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:43.465339    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-kube-rbac-proxy-cm podName:626ec6c0-d616-40ae-bf8b-4e8a927c31ac nodeName:}" failed. No retries permitted until 2026-10-30 00:31:47.465323208 +0000 UTC m=+38.092139036 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jbmfs" (UID: "626ec6c0-d616-40ae-bf8b-4e8a927c31ac") : configmap "kube-rbac-proxy" not found
Oct 30 00:31:43.465486 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:43.465359    2548 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:31:43.465486 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:43.465399    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-metrics-tls\") pod \"dns-default-9ml2g\" (UID: \"b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f\") " pod="openshift-dns/dns-default-9ml2g"
Oct 30 00:31:43.465486 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:43.465409    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-insights-runtime-extractor-tls podName:626ec6c0-d616-40ae-bf8b-4e8a927c31ac nodeName:}" failed. No retries permitted until 2026-10-30 00:31:47.465394374 +0000 UTC m=+38.092210198 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jbmfs" (UID: "626ec6c0-d616-40ae-bf8b-4e8a927c31ac") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:31:43.465486 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:43.465474    2548 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:31:43.465622 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:43.465517    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-metrics-tls podName:b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f nodeName:}" failed. No retries permitted until 2026-10-30 00:31:47.465505954 +0000 UTC m=+38.092321773 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-metrics-tls") pod "dns-default-9ml2g" (UID: "b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f") : secret "dns-default-metrics-tls" not found
Oct 30 00:31:43.566256 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:43.566226    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e552ba30-5b09-48a7-bffb-6a65a094c2bc-metrics-certs\") pod \"network-metrics-daemon-jldmt\" (UID: \"e552ba30-5b09-48a7-bffb-6a65a094c2bc\") " pod="openshift-multus/network-metrics-daemon-jldmt"
Oct 30 00:31:43.566374 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:43.566276    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b0df054-0749-4417-bb8b-b12b3353d25e-original-pull-secret\") pod \"global-pull-secret-syncer-f27vk\" (UID: \"3b0df054-0749-4417-bb8b-b12b3353d25e\") " pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:43.566434 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:43.566382    2548 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 00:31:43.566488 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:43.566458    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e552ba30-5b09-48a7-bffb-6a65a094c2bc-metrics-certs podName:e552ba30-5b09-48a7-bffb-6a65a094c2bc nodeName:}" failed. No retries permitted until 2026-10-30 00:32:15.56643761 +0000 UTC m=+66.193253414 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e552ba30-5b09-48a7-bffb-6a65a094c2bc-metrics-certs") pod "network-metrics-daemon-jldmt" (UID: "e552ba30-5b09-48a7-bffb-6a65a094c2bc") : secret "metrics-daemon-secret" not found
Oct 30 00:31:43.568395 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:43.568374    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b0df054-0749-4417-bb8b-b12b3353d25e-original-pull-secret\") pod \"global-pull-secret-syncer-f27vk\" (UID: \"3b0df054-0749-4417-bb8b-b12b3353d25e\") " pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:43.667046 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:43.667007    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xvts\" (UniqueName: \"kubernetes.io/projected/b3705749-d093-4b49-878d-e4616e81a450-kube-api-access-7xvts\") pod \"network-check-target-wpvjg\" (UID: \"b3705749-d093-4b49-878d-e4616e81a450\") " pod="openshift-network-diagnostics/network-check-target-wpvjg"
Oct 30 00:31:43.669791 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:43.669750    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xvts\" (UniqueName: \"kubernetes.io/projected/b3705749-d093-4b49-878d-e4616e81a450-kube-api-access-7xvts\") pod \"network-check-target-wpvjg\" (UID: \"b3705749-d093-4b49-878d-e4616e81a450\") " pod="openshift-network-diagnostics/network-check-target-wpvjg"
Oct 30 00:31:43.794013 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:43.793926    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:43.800964 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:43.800939    2548 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpvjg"
Oct 30 00:31:44.092799 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:44.089876    2548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f27vk"]
Oct 30 00:31:44.096407 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:44.096376    2548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b0df054_0749_4417_bb8b_b12b3353d25e.slice/crio-9a0705a5c5f0b4f77899aa6fa69d7e239c6d1e7983fc49afc0d77c8886732f74 WatchSource:0}: Error finding container 9a0705a5c5f0b4f77899aa6fa69d7e239c6d1e7983fc49afc0d77c8886732f74: Status 404 returned error can't find the container with id 9a0705a5c5f0b4f77899aa6fa69d7e239c6d1e7983fc49afc0d77c8886732f74
Oct 30 00:31:44.102132 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:44.102108    2548 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wpvjg"]
Oct 30 00:31:44.108473 ip-10-0-135-56 kubenswrapper[2548]: W1030 00:31:44.108448    2548 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3705749_d093_4b49_878d_e4616e81a450.slice/crio-18f770b176e797172e78576db0d907672cf4dc7c54b6d4366fd9eef43baa1f69 WatchSource:0}: Error finding container 18f770b176e797172e78576db0d907672cf4dc7c54b6d4366fd9eef43baa1f69: Status 404 returned error can't find the container with id 18f770b176e797172e78576db0d907672cf4dc7c54b6d4366fd9eef43baa1f69
Oct 30 00:31:44.165203 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:44.165165    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-mln7q" event={"ID":"8ff169ff-dc1a-4344-b19b-adaabe98f20d","Type":"ContainerStarted","Data":"7a9597b0986d0e15285eebee649fb6f052011ed6c5c2620cf55bbff999f0dc27"}
Oct 30 00:31:44.166367 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:44.166335    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wpvjg" event={"ID":"b3705749-d093-4b49-878d-e4616e81a450","Type":"ContainerStarted","Data":"18f770b176e797172e78576db0d907672cf4dc7c54b6d4366fd9eef43baa1f69"}
Oct 30 00:31:44.167333 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:44.167309    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f27vk" event={"ID":"3b0df054-0749-4417-bb8b-b12b3353d25e","Type":"ContainerStarted","Data":"9a0705a5c5f0b4f77899aa6fa69d7e239c6d1e7983fc49afc0d77c8886732f74"}
Oct 30 00:31:44.465763 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:44.465735    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9hslh_30e43c48-92a7-4c68-887c-9aaf7d6d1e1b/node-ca/0.log"
Oct 30 00:31:44.866311 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:44.866223    2548 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jb5sp_a20a733f-4947-4a14-9db0-a5be23cb9ccb/serve-healthcheck-canary/0.log"
Oct 30 00:31:47.500092 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:47.500049    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-metrics-tls\") pod \"dns-default-9ml2g\" (UID: \"b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f\") " pod="openshift-dns/dns-default-9ml2g"
Oct 30 00:31:47.500689 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:47.500112    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:47.500689 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:47.500198    2548 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:47.500689 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:47.500234    2548 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 00:31:47.500689 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:47.500290    2548 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:31:47.500689 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:47.500348    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-insights-runtime-extractor-tls podName:626ec6c0-d616-40ae-bf8b-4e8a927c31ac nodeName:}" failed. No retries permitted until 2026-10-30 00:31:55.500327734 +0000 UTC m=+46.127143540 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jbmfs" (UID: "626ec6c0-d616-40ae-bf8b-4e8a927c31ac") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:31:47.500689 ip-10-0-135-56 kubenswrapper[2548]: E1030 00:31:47.500368    2548 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-metrics-tls podName:b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f nodeName:}" failed. No retries permitted until 2026-10-30 00:31:55.500358357 +0000 UTC m=+46.127174166 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-metrics-tls") pod "dns-default-9ml2g" (UID: "b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f") : secret "dns-default-metrics-tls" not found
Oct 30 00:31:47.500689 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:47.500665    2548 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:48.180429 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:48.180385    2548 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wpvjg" event={"ID":"b3705749-d093-4b49-878d-e4616e81a450","Type":"ContainerStarted","Data":"d99cfeb39a40b2283b714935a14f8ce69047dc12f9348db529994240239ebb77"}
Oct 30 00:31:48.180643 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:48.180478    2548 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wpvjg"
Oct 30 00:31:48.219627 ip-10-0-135-56 kubenswrapper[2548]: I1030 00:31:48.218652    2548 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-mln7q" podStartSLOduration=5.86542692 podStartE2EDuration="8.218628857s" totalImagesPullingTime="2.353201937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:31:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:31:44.199870581 +0000 UTC m=+34.826686408" watchObservedRunningTime="2026-10-30 00:31:48.218628857 +0000 UTC m=+38.845444685"
Oct 30 00:31:49.520974 ip-10-0-135-56 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 00:31:49.537291 ip-10-0-135-56 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 00:31:49.537471 ip-10-0-135-56 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 00:31:49.537526 ip-10-0-135-56 systemd[1]: kubelet.service: Consumed 2.484s CPU time.
Oct 30 00:31:49.551489 ip-10-0-135-56 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 00:31:49.644843 ip-10-0-135-56 kubenswrapper[4393]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:31:49.644843 ip-10-0-135-56 kubenswrapper[4393]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 00:31:49.644843 ip-10-0-135-56 kubenswrapper[4393]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:31:49.644843 ip-10-0-135-56 kubenswrapper[4393]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 00:31:49.647161 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647068    4393 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:31:49.647161 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647161    4393 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:31:49.647161 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647166    4393 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:31:49.647271 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647170    4393 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:31:49.647271 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647173    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:31:49.647271 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647175    4393 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:31:49.647271 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647178    4393 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:31:49.647271 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647181    4393 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:31:49.647271 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647184    4393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:31:49.647271 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647186    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:31:49.647271 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647188    4393 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:31:49.647271 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647191    4393 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:31:49.647271 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647194    4393 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:31:49.647271 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647196    4393 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:31:49.647271 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647205    4393 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:31:49.647271 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647209    4393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:31:49.647271 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647212    4393 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:31:49.647271 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647215    4393 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:31:49.647271 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647218    4393 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:31:49.647271 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647220    4393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:31:49.647271 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647222    4393 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:31:49.647271 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647225    4393 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:31:49.647271 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647227    4393 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:31:49.647790 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647230    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:31:49.647790 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647233    4393 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:31:49.647790 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647235    4393 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:31:49.647790 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647238    4393 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:31:49.647790 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647240    4393 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:31:49.647790 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647243    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:31:49.647790 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647246    4393 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:31:49.647790 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647248    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:31:49.647790 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647251    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:31:49.647790 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647253    4393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:31:49.647790 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647256    4393 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:31:49.647790 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647259    4393 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:31:49.647790 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647261    4393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:31:49.647790 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647264    4393 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:31:49.647790 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647267    4393 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:31:49.647790 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647269    4393 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:31:49.647790 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647272    4393 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:31:49.647790 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647275    4393 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:31:49.647790 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647278    4393 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:31:49.647790 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647280    4393 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:31:49.648289 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647283    4393 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:31:49.648289 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647285    4393 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:31:49.648289 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647290    4393 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:31:49.648289 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647295    4393 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:31:49.648289 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647299    4393 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:31:49.648289 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647302    4393 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:31:49.648289 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647305    4393 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:31:49.648289 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647307    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:31:49.648289 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647310    4393 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:31:49.648289 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647312    4393 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:31:49.648289 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647315    4393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:31:49.648289 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647318    4393 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:31:49.648289 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647320    4393 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:31:49.648289 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647322    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:31:49.648289 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647325    4393 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:31:49.648289 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647328    4393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:31:49.648289 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647331    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:31:49.648289 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647333    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:31:49.648289 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647336    4393 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:31:49.648753 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647340    4393 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:31:49.648753 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647342    4393 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:31:49.648753 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647345    4393 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:31:49.648753 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647348    4393 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:31:49.648753 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647351    4393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:31:49.648753 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647354    4393 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:31:49.648753 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647357    4393 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:31:49.648753 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647360    4393 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:31:49.648753 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647362    4393 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:31:49.648753 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647365    4393 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:31:49.648753 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647367    4393 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:31:49.648753 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647370    4393 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:31:49.648753 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647372    4393 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:31:49.648753 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647375    4393 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:31:49.648753 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647377    4393 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:31:49.648753 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647380    4393 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:31:49.648753 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647386    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:31:49.648753 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647389    4393 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:31:49.648753 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647391    4393 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:31:49.648753 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647394    4393 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:31:49.649273 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647397    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:31:49.649273 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647400    4393 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:31:49.649273 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647402    4393 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:31:49.649273 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647404    4393 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:31:49.649273 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647407    4393 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:31:49.649273 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647410    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:31:49.649273 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647413    4393 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:31:49.649273 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647417    4393 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:31:49.649273 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647420    4393 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:31:49.649273 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647422    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:31:49.649273 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647425    4393 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:31:49.649273 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647428    4393 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:31:49.649273 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647430    4393 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:31:49.649273 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647433    4393 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:31:49.649273 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647436    4393 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:31:49.649273 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647438    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:31:49.649273 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647441    4393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:31:49.649273 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647443    4393 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:31:49.649273 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647446    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:31:49.649742 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647448    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:31:49.649742 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647451    4393 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:31:49.649742 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647454    4393 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:31:49.649742 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647456    4393 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:31:49.649742 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.647459    4393 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:31:49.649742 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648302    4393 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:31:49.649742 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648310    4393 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:31:49.649742 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648313    4393 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:31:49.649742 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648316    4393 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:31:49.649742 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648319    4393 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:31:49.649742 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648331    4393 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:31:49.649742 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648334    4393 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:31:49.649742 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648336    4393 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:31:49.649742 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648339    4393 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:31:49.649742 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648342    4393 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:31:49.649742 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648344    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:31:49.649742 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648347    4393 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:31:49.649742 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648350    4393 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:31:49.649742 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648353    4393 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:31:49.649742 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648355    4393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:31:49.650267 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648358    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:31:49.650267 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648361    4393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:31:49.650267 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648364    4393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:31:49.650267 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648366    4393 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:31:49.650267 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648369    4393 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:31:49.650267 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648372    4393 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:31:49.650267 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648375    4393 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:31:49.650267 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648377    4393 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:31:49.650267 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648380    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:31:49.650267 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648382    4393 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:31:49.650267 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648385    4393 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:31:49.650267 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648387    4393 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:31:49.650267 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648390    4393 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:31:49.650267 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648392    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:31:49.650267 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648394    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:31:49.650267 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648397    4393 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:31:49.650267 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648399    4393 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:31:49.650267 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648402    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:31:49.650267 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648404    4393 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:31:49.650267 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648407    4393 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:31:49.650769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648409    4393 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:31:49.650769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648418    4393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:31:49.650769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648421    4393 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:31:49.650769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648424    4393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:31:49.650769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648426    4393 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:31:49.650769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648429    4393 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:31:49.650769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648431    4393 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:31:49.650769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648434    4393 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:31:49.650769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648436    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:31:49.650769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648439    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:31:49.650769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648442    4393 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:31:49.650769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648444    4393 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:31:49.650769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648447    4393 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:31:49.650769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648449    4393 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:31:49.650769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648452    4393 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:31:49.650769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648455    4393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:31:49.650769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648457    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:31:49.650769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648460    4393 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:31:49.650769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648463    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:31:49.651247 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648465    4393 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:31:49.651247 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648468    4393 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:31:49.651247 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648470    4393 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:31:49.651247 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648473    4393 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:31:49.651247 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648475    4393 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:31:49.651247 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648478    4393 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:31:49.651247 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648480    4393 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:31:49.651247 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648483    4393 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:31:49.651247 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648485    4393 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:31:49.651247 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648488    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:31:49.651247 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648490    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:31:49.651247 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648493    4393 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:31:49.651247 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648495    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:31:49.651247 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648498    4393 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:31:49.651247 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648500    4393 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:31:49.651247 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648504    4393 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:31:49.651247 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648506    4393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:31:49.651247 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648510    4393 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:31:49.651247 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648513    4393 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:31:49.651247 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648515    4393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:31:49.651769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648518    4393 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:31:49.651769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648520    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:31:49.651769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648524    4393 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:31:49.651769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648526    4393 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:31:49.651769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648529    4393 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:31:49.651769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648533    4393 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:31:49.651769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648536    4393 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:31:49.651769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648540    4393 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:31:49.651769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648542    4393 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:31:49.651769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648545    4393 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:31:49.651769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648549    4393 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:31:49.651769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648551    4393 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:31:49.651769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648554    4393 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:31:49.651769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648556    4393 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:31:49.651769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648559    4393 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:31:49.651769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648561    4393 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:31:49.651769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648564    4393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:31:49.651769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648567    4393 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:31:49.651769 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648569    4393 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:31:49.652251 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648571    4393 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:31:49.652251 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648574    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:31:49.652251 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648579    4393 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:31:49.652251 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648583    4393 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:31:49.652251 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648587    4393 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:31:49.652251 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648589    4393 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:31:49.652251 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648592    4393 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:31:49.652251 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648594    4393 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:31:49.652251 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648597    4393 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:31:49.652251 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648599    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:31:49.652251 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648602    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:31:49.652251 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648605    4393 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:31:49.652251 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.648608    4393 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:31:49.652251 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649161    4393 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:31:49.652251 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649168    4393 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:31:49.652251 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649171    4393 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:31:49.652251 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649174    4393 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:31:49.652251 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649177    4393 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:31:49.652251 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649179    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:31:49.652727 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649182    4393 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:31:49.652727 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649186    4393 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:31:49.652727 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649189    4393 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:31:49.652727 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649191    4393 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:31:49.652727 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649194    4393 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:31:49.652727 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649197    4393 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:31:49.652727 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649200    4393 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:31:49.652727 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649203    4393 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:31:49.652727 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649207    4393 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:31:49.652727 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649210    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:31:49.652727 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649213    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:31:49.652727 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649216    4393 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:31:49.652727 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649219    4393 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:31:49.652727 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649222    4393 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:31:49.652727 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649225    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:31:49.652727 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649228    4393 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:31:49.652727 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649231    4393 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:31:49.652727 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649233    4393 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:31:49.652727 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649236    4393 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:31:49.652727 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649238    4393 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:31:49.653228 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649241    4393 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:31:49.653228 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649246    4393 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:31:49.653228 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649249    4393 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:31:49.653228 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649252    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:31:49.653228 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649255    4393 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:31:49.653228 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649258    4393 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:31:49.653228 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649261    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:31:49.653228 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649263    4393 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:31:49.653228 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649266    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:31:49.653228 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649268    4393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:31:49.653228 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649271    4393 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:31:49.653228 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649274    4393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:31:49.653228 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649276    4393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:31:49.653228 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649279    4393 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:31:49.653228 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649282    4393 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:31:49.653228 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649285    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:31:49.653228 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649287    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:31:49.653228 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649290    4393 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:31:49.653228 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649292    4393 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:31:49.653699 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649294    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:31:49.653699 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649297    4393 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:31:49.653699 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649299    4393 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:31:49.653699 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649302    4393 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:31:49.653699 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649304    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:31:49.653699 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649307    4393 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:31:49.653699 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649309    4393 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:31:49.653699 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649311    4393 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:31:49.653699 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649314    4393 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:31:49.653699 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649316    4393 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:31:49.653699 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649319    4393 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:31:49.653699 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649321    4393 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:31:49.653699 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649323    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:31:49.653699 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649326    4393 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:31:49.653699 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649330    4393 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:31:49.653699 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649332    4393 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:31:49.653699 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649334    4393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:31:49.653699 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649337    4393 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:31:49.653699 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649340    4393 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:31:49.654172 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649343    4393 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:31:49.654172 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649346    4393 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:31:49.654172 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649349    4393 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:31:49.654172 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649351    4393 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:31:49.654172 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649354    4393 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:31:49.654172 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649356    4393 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:31:49.654172 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649359    4393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:31:49.654172 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649363    4393 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:31:49.654172 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649366    4393 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:31:49.654172 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649368    4393 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:31:49.654172 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649371    4393 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:31:49.654172 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649373    4393 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:31:49.654172 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649376    4393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:31:49.654172 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649378    4393 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:31:49.654172 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649381    4393 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:31:49.654172 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649384    4393 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:31:49.654172 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649388    4393 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:31:49.654172 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649391    4393 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:31:49.654172 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649394    4393 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:31:49.654642 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649398    4393 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:31:49.654642 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649400    4393 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:31:49.654642 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649403    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:31:49.654642 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649405    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:31:49.654642 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649408    4393 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:31:49.654642 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649410    4393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:31:49.654642 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649413    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:31:49.654642 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649415    4393 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:31:49.654642 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649419    4393 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:31:49.654642 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649422    4393 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:31:49.654642 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649424    4393 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:31:49.654642 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649427    4393 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:31:49.654642 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649431    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:31:49.654642 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649435    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:31:49.654642 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649437    4393 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:31:49.654642 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649440    4393 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:31:49.654642 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649442    4393 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:31:49.654642 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649445    4393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:31:49.654642 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649447    4393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:31:49.654642 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649451    4393 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:31:49.655154 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649453    4393 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:31:49.655154 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649456    4393 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:31:49.655154 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.649458    4393 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:31:49.655154 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649569    4393 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 00:31:49.655154 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649577    4393 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 00:31:49.655154 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649584    4393 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 00:31:49.655154 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649589    4393 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 00:31:49.655154 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649593    4393 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 00:31:49.655154 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649596    4393 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 00:31:49.655154 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649600    4393 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 00:31:49.655154 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649605    4393 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 00:31:49.655154 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649608    4393 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 00:31:49.655154 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649611    4393 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 00:31:49.655154 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649615    4393 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 00:31:49.655154 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649618    4393 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 00:31:49.655154 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649621    4393 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 00:31:49.655154 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649624    4393 flags.go:64] FLAG: --cgroup-root=""
Oct 30 00:31:49.655154 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649627    4393 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 00:31:49.655154 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649629    4393 flags.go:64] FLAG: --client-ca-file=""
Oct 30 00:31:49.655154 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649632    4393 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 00:31:49.655154 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649637    4393 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 00:31:49.655154 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649642    4393 flags.go:64] FLAG: --cluster-domain=""
Oct 30 00:31:49.655154 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649645    4393 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 00:31:49.655154 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649648    4393 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 00:31:49.655727 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649654    4393 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 00:31:49.655727 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649657    4393 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 00:31:49.655727 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649662    4393 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 00:31:49.655727 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649665    4393 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 00:31:49.655727 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649668    4393 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 00:31:49.655727 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649672    4393 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 00:31:49.655727 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649675    4393 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 00:31:49.655727 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649678    4393 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 00:31:49.655727 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649681    4393 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 00:31:49.655727 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649684    4393 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 00:31:49.655727 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649687    4393 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 00:31:49.655727 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649691    4393 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 00:31:49.655727 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649694    4393 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 00:31:49.655727 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649697    4393 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 00:31:49.655727 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649700    4393 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 00:31:49.655727 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649703    4393 flags.go:64] FLAG: --enable-server="true"
Oct 30 00:31:49.655727 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649706    4393 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 00:31:49.655727 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649710    4393 flags.go:64] FLAG: --event-burst="100"
Oct 30 00:31:49.655727 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649714    4393 flags.go:64] FLAG: --event-qps="50"
Oct 30 00:31:49.655727 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649716    4393 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 00:31:49.655727 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649720    4393 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 00:31:49.655727 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649722    4393 flags.go:64] FLAG: --eviction-hard=""
Oct 30 00:31:49.655727 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649728    4393 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 00:31:49.655727 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649731    4393 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 00:31:49.656440 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649734    4393 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 00:31:49.656440 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649737    4393 flags.go:64] FLAG: --eviction-soft=""
Oct 30 00:31:49.656440 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649740    4393 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 00:31:49.656440 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649743    4393 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 00:31:49.656440 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649747    4393 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 00:31:49.656440 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649750    4393 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 00:31:49.656440 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649753    4393 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 00:31:49.656440 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649756    4393 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 00:31:49.656440 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649761    4393 flags.go:64] FLAG: --feature-gates=""
Oct 30 00:31:49.656440 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649768    4393 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 00:31:49.656440 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649771    4393 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 00:31:49.656440 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649789    4393 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 00:31:49.656440 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649792    4393 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 00:31:49.656440 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649796    4393 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 00:31:49.656440 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649799    4393 flags.go:64] FLAG: --help="false"
Oct 30 00:31:49.656440 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649802    4393 flags.go:64] FLAG: --hostname-override="ip-10-0-135-56.ec2.internal"
Oct 30 00:31:49.656440 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649805    4393 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 00:31:49.656440 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649808    4393 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 00:31:49.656440 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649811    4393 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 00:31:49.656440 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649814    4393 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 00:31:49.656440 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649817    4393 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 00:31:49.656440 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649820    4393 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 00:31:49.656440 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649823    4393 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 00:31:49.657015 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649826    4393 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 00:31:49.657015 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649829    4393 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 00:31:49.657015 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649831    4393 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 00:31:49.657015 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649835    4393 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 00:31:49.657015 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649837    4393 flags.go:64] FLAG: --kube-reserved=""
Oct 30 00:31:49.657015 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649840    4393 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 00:31:49.657015 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649844    4393 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 00:31:49.657015 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649847    4393 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 00:31:49.657015 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649849    4393 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 00:31:49.657015 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649852    4393 flags.go:64] FLAG: --lock-file=""
Oct 30 00:31:49.657015 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649855    4393 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 00:31:49.657015 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649857    4393 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 00:31:49.657015 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649860    4393 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 00:31:49.657015 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649868    4393 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 00:31:49.657015 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649871    4393 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 00:31:49.657015 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649873    4393 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 00:31:49.657015 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649877    4393 flags.go:64] FLAG: --logging-format="text"
Oct 30 00:31:49.657015 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649881    4393 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 00:31:49.657015 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649884    4393 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 00:31:49.657015 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649887    4393 flags.go:64] FLAG: --manifest-url=""
Oct 30 00:31:49.657015 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649890    4393 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 00:31:49.657015 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649895    4393 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 00:31:49.657015 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649898    4393 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 00:31:49.657015 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649902    4393 flags.go:64] FLAG: --max-pods="110"
Oct 30 00:31:49.657015 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649905    4393 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 00:31:49.657626 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649908    4393 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 00:31:49.657626 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649911    4393 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 00:31:49.657626 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649914    4393 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 00:31:49.657626 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649917    4393 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 00:31:49.657626 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649920    4393 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 00:31:49.657626 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649923    4393 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 00:31:49.657626 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649930    4393 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 00:31:49.657626 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649933    4393 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 00:31:49.657626 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649936    4393 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 00:31:49.657626 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649939    4393 flags.go:64] FLAG: --pod-cidr=""
Oct 30 00:31:49.657626 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649942    4393 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 00:31:49.657626 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649945    4393 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 00:31:49.657626 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649948    4393 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 00:31:49.657626 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649951    4393 flags.go:64] FLAG: --port="10250"
Oct 30 00:31:49.657626 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649954    4393 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 00:31:49.657626 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649957    4393 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-000c194909efd78be"
Oct 30 00:31:49.657626 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649960    4393 flags.go:64] FLAG: --qos-reserved=""
Oct 30 00:31:49.657626 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649962    4393 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 00:31:49.657626 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649965    4393 flags.go:64] FLAG: --register-node="true"
Oct 30 00:31:49.657626 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649968    4393 flags.go:64] FLAG: --register-with-taints=""
Oct 30 00:31:49.657626 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649974    4393 flags.go:64] FLAG: --registry-burst="10"
Oct 30 00:31:49.657626 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649977    4393 flags.go:64] FLAG: --registry-qps="5"
Oct 30 00:31:49.657626 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649979    4393 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 00:31:49.657626 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649982    4393 flags.go:64] FLAG: --reserved-memory=""
Oct 30 00:31:49.657626 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649988    4393 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 00:31:49.657626 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649991    4393 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 00:31:49.658393 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649994    4393 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 00:31:49.658393 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.649997    4393 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 00:31:49.658393 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.650000    4393 flags.go:64] FLAG: --runonce="false"
Oct 30 00:31:49.658393 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.650003    4393 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 00:31:49.658393 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.650006    4393 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 00:31:49.658393 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.650009    4393 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 00:31:49.658393 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.650012    4393 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 00:31:49.658393 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.650015    4393 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 00:31:49.658393 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.650018    4393 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 00:31:49.658393 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.650022    4393 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 00:31:49.658393 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.650025    4393 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 00:31:49.658393 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.650028    4393 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 00:31:49.658393 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.650030    4393 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 00:31:49.658393 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.650033    4393 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 00:31:49.658393 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.650036    4393 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 00:31:49.658393 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.650039    4393 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 00:31:49.658393 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.650042    4393 flags.go:64] FLAG: --system-cgroups=""
Oct 30 00:31:49.658393 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.650045    4393 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 00:31:49.658393 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.650050    4393 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 00:31:49.658393 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.650053    4393 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 00:31:49.658393 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.650056    4393 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 00:31:49.658393 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.650060    4393 flags.go:64] FLAG: --tls-min-version=""
Oct 30 00:31:49.658393 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.650063    4393 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 00:31:49.658393 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.650066    4393 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 00:31:49.658393 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.650069    4393 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 00:31:49.659118 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.650072    4393 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 00:31:49.659118 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.650076    4393 flags.go:64] FLAG: --v="2"
Oct 30 00:31:49.659118 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.650081    4393 flags.go:64] FLAG: --version="false"
Oct 30 00:31:49.659118 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.650085    4393 flags.go:64] FLAG: --vmodule=""
Oct 30 00:31:49.659118 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.650089    4393 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 00:31:49.659118 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.650093    4393 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 00:31:49.659118 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650235    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:31:49.659118 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650238    4393 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:31:49.659118 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650241    4393 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:31:49.659118 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650244    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:31:49.659118 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650246    4393 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:31:49.659118 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650250    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:31:49.659118 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650252    4393 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:31:49.659118 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650255    4393 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:31:49.659118 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650258    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:31:49.659118 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650260    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:31:49.659118 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650263    4393 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:31:49.659118 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650265    4393 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:31:49.659118 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650268    4393 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:31:49.659118 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650270    4393 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:31:49.659118 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650273    4393 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:31:49.659927 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650276    4393 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:31:49.659927 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650279    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:31:49.659927 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650281    4393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:31:49.659927 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650284    4393 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:31:49.659927 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650286    4393 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:31:49.659927 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650289    4393 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:31:49.659927 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650292    4393 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:31:49.659927 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650296    4393 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:31:49.659927 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650299    4393 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:31:49.659927 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650302    4393 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:31:49.659927 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650305    4393 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:31:49.659927 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650308    4393 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:31:49.659927 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650311    4393 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:31:49.659927 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650314    4393 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:31:49.659927 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650316    4393 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:31:49.659927 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650319    4393 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:31:49.659927 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650322    4393 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:31:49.659927 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650325    4393 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:31:49.659927 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650328    4393 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:31:49.660678 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650330    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:31:49.660678 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650333    4393 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:31:49.660678 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650335    4393 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:31:49.660678 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650338    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:31:49.660678 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650341    4393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:31:49.660678 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650343    4393 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:31:49.660678 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650346    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:31:49.660678 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650349    4393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:31:49.660678 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650351    4393 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:31:49.660678 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650354    4393 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:31:49.660678 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650356    4393 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:31:49.660678 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650359    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:31:49.660678 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650361    4393 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:31:49.660678 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650364    4393 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:31:49.660678 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650366    4393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:31:49.660678 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650369    4393 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:31:49.660678 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650371    4393 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:31:49.660678 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650374    4393 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:31:49.660678 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650376    4393 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:31:49.660678 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650379    4393 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:31:49.661494 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650381    4393 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:31:49.661494 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650384    4393 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:31:49.661494 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650387    4393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:31:49.661494 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650389    4393 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:31:49.661494 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650392    4393 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:31:49.661494 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650394    4393 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:31:49.661494 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650397    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:31:49.661494 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650399    4393 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:31:49.661494 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650402    4393 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:31:49.661494 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650405    4393 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:31:49.661494 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650408    4393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:31:49.661494 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650411    4393 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:31:49.661494 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650413    4393 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:31:49.661494 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650415    4393 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:31:49.661494 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650418    4393 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:31:49.661494 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650421    4393 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:31:49.661494 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650423    4393 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:31:49.661494 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650426    4393 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:31:49.661494 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650429    4393 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:31:49.661494 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650431    4393 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:31:49.662084 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650434    4393 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:31:49.662084 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650436    4393 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:31:49.662084 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650439    4393 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:31:49.662084 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650441    4393 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:31:49.662084 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650444    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:31:49.662084 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650446    4393 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:31:49.662084 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650449    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:31:49.662084 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650451    4393 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:31:49.662084 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650454    4393 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:31:49.662084 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650456    4393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:31:49.662084 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650459    4393 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:31:49.662084 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650461    4393 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:31:49.662084 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650464    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:31:49.662084 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650466    4393 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:31:49.662084 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650469    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:31:49.662084 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650472    4393 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:31:49.662084 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650474    4393 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:31:49.662084 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650476    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:31:49.662084 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650479    4393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:31:49.662576 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650481    4393 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:31:49.662576 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650484    4393 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:31:49.662576 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650488    4393 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:31:49.662576 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650490    4393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:31:49.662576 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650493    4393 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:31:49.662576 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650495    4393 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:31:49.662576 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650498    4393 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:31:49.662576 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650500    4393 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:31:49.662576 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650504    4393 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:31:49.662576 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650507    4393 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:31:49.662576 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650510    4393 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:31:49.662576 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650513    4393 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:31:49.662576 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650516    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:31:49.662576 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.650551    4393 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:31:49.662576 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650589    4393 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:31:49.662576 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650593    4393 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:31:49.662993 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650597    4393 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:31:49.662993 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650600    4393 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:31:49.662993 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650603    4393 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:31:49.662993 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650605    4393 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:31:49.662993 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650608    4393 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:31:49.662993 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650610    4393 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:31:49.662993 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650613    4393 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:31:49.662993 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650615    4393 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:31:49.662993 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650618    4393 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:31:49.662993 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650620    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:31:49.662993 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650623    4393 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:31:49.662993 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650626    4393 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:31:49.662993 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650628    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:31:49.662993 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650631    4393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:31:49.662993 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650633    4393 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:31:49.662993 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650636    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:31:49.662993 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650638    4393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:31:49.662993 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650641    4393 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:31:49.662993 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650643    4393 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:31:49.662993 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650647    4393 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:31:49.663505 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650650    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:31:49.663505 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650652    4393 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:31:49.663505 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650655    4393 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:31:49.663505 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650657    4393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:31:49.663505 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650660    4393 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:31:49.663505 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650663    4393 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:31:49.663505 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650665    4393 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:31:49.663505 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650668    4393 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:31:49.663505 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650670    4393 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:31:49.663505 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650673    4393 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:31:49.663505 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650676    4393 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:31:49.663505 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650678    4393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:31:49.663505 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650681    4393 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:31:49.663505 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650683    4393 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:31:49.663505 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650686    4393 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:31:49.663505 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650688    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:31:49.663505 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650691    4393 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:31:49.663505 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650694    4393 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:31:49.663505 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650696    4393 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:31:49.663505 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650698    4393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:31:49.664161 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650703    4393 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:31:49.664161 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650705    4393 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:31:49.664161 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650708    4393 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:31:49.664161 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650710    4393 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:31:49.664161 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650712    4393 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:31:49.664161 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650715    4393 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:31:49.664161 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650717    4393 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:31:49.664161 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650720    4393 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:31:49.664161 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650722    4393 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:31:49.664161 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650725    4393 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:31:49.664161 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650728    4393 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:31:49.664161 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650731    4393 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:31:49.664161 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650734    4393 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:31:49.664161 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650736    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:31:49.664161 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650739    4393 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:31:49.664161 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650741    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:31:49.664161 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650744    4393 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:31:49.664161 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650747    4393 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:31:49.664161 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650749    4393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:31:49.664161 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650751    4393 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:31:49.664695 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650754    4393 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:31:49.664695 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650757    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:31:49.664695 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650759    4393 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:31:49.664695 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650762    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:31:49.664695 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650766    4393 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:31:49.664695 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650768    4393 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:31:49.664695 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650771    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:31:49.664695 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650787    4393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:31:49.664695 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650790    4393 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:31:49.664695 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650793    4393 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:31:49.664695 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650795    4393 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:31:49.664695 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650798    4393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:31:49.664695 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650802    4393 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:31:49.664695 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650804    4393 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:31:49.664695 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650807    4393 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:31:49.664695 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650809    4393 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:31:49.664695 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650813    4393 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:31:49.664695 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650816    4393 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:31:49.664695 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650819    4393 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:31:49.665234 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650822    4393 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:31:49.665234 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650825    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:31:49.665234 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650827    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:31:49.665234 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650829    4393 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:31:49.665234 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650833    4393 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:31:49.665234 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650835    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:31:49.665234 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650838    4393 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:31:49.665234 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650840    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:31:49.665234 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650843    4393 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:31:49.665234 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650845    4393 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:31:49.665234 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650848    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:31:49.665234 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650850    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:31:49.665234 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650853    4393 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:31:49.665234 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650855    4393 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:31:49.665234 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650858    4393 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:31:49.665234 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650861    4393 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:31:49.665234 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650863    4393 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:31:49.665234 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650866    4393 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:31:49.665234 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650868    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:31:49.665689 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650870    4393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:31:49.665689 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650873    4393 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:31:49.665689 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650875    4393 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:31:49.665689 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650877    4393 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:31:49.665689 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650880    4393 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:31:49.665689 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.650885    4393 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:31:49.665689 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.654853    4393 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 00:31:49.665689 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.654869    4393 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 00:31:49.665689 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.654890    4393 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 00:31:49.665689 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.654894    4393 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 00:31:49.665689 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.654956    4393 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:31:49.665689 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.654961    4393 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:31:49.665689 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.654964    4393 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:31:49.665689 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.654967    4393 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:31:49.665689 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.654970    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:31:49.665689 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.654972    4393 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:31:49.665689 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.654975    4393 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:31:49.665689 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.654978    4393 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:31:49.665689 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.654980    4393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:31:49.666170 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.654983    4393 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:31:49.666170 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.654985    4393 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:31:49.666170 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.654988    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:31:49.666170 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.654990    4393 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:31:49.666170 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.654993    4393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:31:49.666170 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.654995    4393 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:31:49.666170 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.654998    4393 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:31:49.666170 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655000    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:31:49.666170 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655003    4393 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:31:49.666170 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655006    4393 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:31:49.666170 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655008    4393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:31:49.666170 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655011    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:31:49.666170 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655014    4393 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:31:49.666170 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655016    4393 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:31:49.666170 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655019    4393 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:31:49.666170 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655021    4393 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:31:49.666170 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655024    4393 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:31:49.666170 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655027    4393 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:31:49.666170 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655030    4393 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:31:49.666170 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655033    4393 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:31:49.666654 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655035    4393 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:31:49.666654 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655038    4393 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:31:49.666654 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655040    4393 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:31:49.666654 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655043    4393 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:31:49.666654 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655045    4393 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:31:49.666654 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655048    4393 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:31:49.666654 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655050    4393 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:31:49.666654 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655053    4393 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:31:49.666654 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655058    4393 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:31:49.666654 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655061    4393 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:31:49.666654 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655065    4393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:31:49.666654 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655068    4393 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:31:49.666654 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655070    4393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:31:49.666654 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655073    4393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:31:49.666654 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655075    4393 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:31:49.666654 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655078    4393 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:31:49.666654 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655080    4393 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:31:49.666654 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655083    4393 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:31:49.666654 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655085    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:31:49.667153 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655088    4393 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:31:49.667153 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655090    4393 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:31:49.667153 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655093    4393 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:31:49.667153 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655096    4393 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:31:49.667153 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655098    4393 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:31:49.667153 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655101    4393 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:31:49.667153 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655104    4393 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:31:49.667153 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655107    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:31:49.667153 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655109    4393 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:31:49.667153 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655113    4393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:31:49.667153 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655115    4393 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:31:49.667153 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655118    4393 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:31:49.667153 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655120    4393 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:31:49.667153 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655123    4393 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:31:49.667153 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655126    4393 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:31:49.667153 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655128    4393 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:31:49.667153 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655131    4393 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:31:49.667153 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655134    4393 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:31:49.667153 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655136    4393 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:31:49.667153 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655139    4393 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:31:49.667680 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655141    4393 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:31:49.667680 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655144    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:31:49.667680 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655146    4393 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:31:49.667680 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655163    4393 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:31:49.667680 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655167    4393 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:31:49.667680 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655170    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:31:49.667680 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655173    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:31:49.667680 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655176    4393 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:31:49.667680 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655179    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:31:49.667680 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655182    4393 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:31:49.667680 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655185    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:31:49.667680 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655187    4393 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:31:49.667680 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655190    4393 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:31:49.667680 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655192    4393 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:31:49.667680 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655196    4393 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:31:49.667680 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655199    4393 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:31:49.667680 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655202    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:31:49.667680 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655205    4393 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:31:49.667680 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655208    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:31:49.668244 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655210    4393 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:31:49.668244 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655213    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:31:49.668244 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655215    4393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:31:49.668244 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655218    4393 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:31:49.668244 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655220    4393 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:31:49.668244 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655223    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:31:49.668244 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655225    4393 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:31:49.668244 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655228    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:31:49.668244 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655230    4393 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:31:49.668244 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655233    4393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:31:49.668244 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655235    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:31:49.668244 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655238    4393 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:31:49.668244 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655240    4393 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:31:49.668244 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655243    4393 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:31:49.668244 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655245    4393 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:31:49.668244 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655248    4393 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:31:49.668244 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655250    4393 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:31:49.668244 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655252    4393 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:31:49.668244 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655255    4393 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:31:49.668721 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.655285    4393 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:31:49.668721 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655425    4393 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:31:49.668721 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655430    4393 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:31:49.668721 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655435    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:31:49.668721 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655438    4393 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:31:49.668721 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655441    4393 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:31:49.668721 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655445    4393 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:31:49.668721 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655447    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:31:49.668721 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655451    4393 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:31:49.668721 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655454    4393 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:31:49.668721 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655457    4393 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:31:49.668721 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655460    4393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:31:49.668721 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655463    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:31:49.668721 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655465    4393 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:31:49.668721 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655468    4393 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:31:49.668721 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655470    4393 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:31:49.669136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655473    4393 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:31:49.669136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655475    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:31:49.669136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655478    4393 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:31:49.669136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655480    4393 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:31:49.669136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655483    4393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:31:49.669136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655485    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:31:49.669136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655487    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:31:49.669136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655490    4393 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:31:49.669136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655492    4393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:31:49.669136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655495    4393 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:31:49.669136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655497    4393 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:31:49.669136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655500    4393 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:31:49.669136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655502    4393 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:31:49.669136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655504    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:31:49.669136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655507    4393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:31:49.669136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655510    4393 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:31:49.669136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655515    4393 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:31:49.669136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655518    4393 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:31:49.669597 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655521    4393 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:31:49.669597 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655523    4393 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:31:49.669597 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655526    4393 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:31:49.669597 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655528    4393 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:31:49.669597 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655531    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:31:49.669597 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655533    4393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:31:49.669597 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655536    4393 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:31:49.669597 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655538    4393 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:31:49.669597 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655541    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:31:49.669597 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655543    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:31:49.669597 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655546    4393 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:31:49.669597 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655548    4393 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:31:49.669597 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655551    4393 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:31:49.669597 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655553    4393 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:31:49.669597 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655556    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:31:49.669597 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655558    4393 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:31:49.669597 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655561    4393 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:31:49.669597 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655563    4393 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:31:49.669597 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655565    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:31:49.669597 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655568    4393 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:31:49.670175 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655570    4393 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:31:49.670175 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655573    4393 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:31:49.670175 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655575    4393 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:31:49.670175 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655577    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:31:49.670175 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655580    4393 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:31:49.670175 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655582    4393 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:31:49.670175 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655584    4393 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:31:49.670175 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655587    4393 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:31:49.670175 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655589    4393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:31:49.670175 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655592    4393 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:31:49.670175 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655597    4393 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:31:49.670175 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655600    4393 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:31:49.670175 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655602    4393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:31:49.670175 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655605    4393 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:31:49.670175 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655608    4393 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:31:49.670175 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655610    4393 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:31:49.670175 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655612    4393 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:31:49.670175 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655615    4393 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:31:49.670175 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655617    4393 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:31:49.670175 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655620    4393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:31:49.670687 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655622    4393 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:31:49.670687 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655624    4393 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:31:49.670687 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655627    4393 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:31:49.670687 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655630    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:31:49.670687 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655633    4393 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:31:49.670687 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655635    4393 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:31:49.670687 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655638    4393 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:31:49.670687 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655641    4393 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:31:49.670687 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655643    4393 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:31:49.670687 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655646    4393 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:31:49.670687 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655648    4393 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:31:49.670687 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655651    4393 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:31:49.670687 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655653    4393 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:31:49.670687 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655656    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:31:49.670687 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655658    4393 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:31:49.670687 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655660    4393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:31:49.670687 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655663    4393 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:31:49.670687 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655665    4393 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:31:49.670687 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655668    4393 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:31:49.670687 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655671    4393 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:31:49.671208 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655673    4393 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:31:49.671208 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655676    4393 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:31:49.671208 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655679    4393 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:31:49.671208 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655682    4393 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:31:49.671208 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655684    4393 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:31:49.671208 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655687    4393 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:31:49.671208 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655689    4393 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:31:49.671208 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655691    4393 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:31:49.671208 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655694    4393 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:31:49.671208 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655696    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:31:49.671208 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655699    4393 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:31:49.671208 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655701    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:31:49.671208 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655704    4393 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:31:49.671208 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.655726    4393 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 00:31:49.671208 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655759    4393 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 00:31:49.671208 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655764    4393 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 00:31:49.671609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655768    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 00:31:49.671609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655771    4393 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 00:31:49.671609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655790    4393 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 00:31:49.671609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655795    4393 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 00:31:49.671609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655799    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 00:31:49.671609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655804    4393 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 00:31:49.671609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655808    4393 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 00:31:49.671609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655810    4393 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 00:31:49.671609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655814    4393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 00:31:49.671609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655816    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 00:31:49.671609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655819    4393 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 00:31:49.671609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655821    4393 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 00:31:49.671609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655823    4393 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 00:31:49.671609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655826    4393 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 00:31:49.671609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655828    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 00:31:49.671609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655831    4393 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 00:31:49.671609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655833    4393 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 00:31:49.671609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655836    4393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 00:31:49.671609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655840    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 00:31:49.671609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655843    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 00:31:49.672201 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655845    4393 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 00:31:49.672201 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655848    4393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 00:31:49.672201 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655850    4393 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 00:31:49.672201 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655853    4393 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 00:31:49.672201 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655855    4393 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 00:31:49.672201 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655858    4393 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 00:31:49.672201 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655860    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 00:31:49.672201 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655863    4393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 00:31:49.672201 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655867    4393 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 00:31:49.672201 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655871    4393 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 00:31:49.672201 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655873    4393 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 00:31:49.672201 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655876    4393 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 00:31:49.672201 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655878    4393 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 00:31:49.672201 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655881    4393 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 00:31:49.672201 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655884    4393 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 00:31:49.672201 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655887    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 00:31:49.672201 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655889    4393 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 00:31:49.672201 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655891    4393 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 00:31:49.672649 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655894    4393 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 00:31:49.672649 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655896    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 00:31:49.672649 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655899    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 00:31:49.672649 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655901    4393 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 00:31:49.672649 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655904    4393 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 00:31:49.672649 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655906    4393 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 00:31:49.672649 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655908    4393 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 00:31:49.672649 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655911    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 00:31:49.672649 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655913    4393 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 00:31:49.672649 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655916    4393 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 00:31:49.672649 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655918    4393 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 00:31:49.672649 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655920    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 00:31:49.672649 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655927    4393 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 00:31:49.672649 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655929    4393 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 00:31:49.672649 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655932    4393 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 00:31:49.672649 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655934    4393 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 00:31:49.672649 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655937    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 00:31:49.672649 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655939    4393 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 00:31:49.672649 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655942    4393 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 00:31:49.672649 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655944    4393 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 00:31:49.673136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655947    4393 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 00:31:49.673136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655950    4393 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 00:31:49.673136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655952    4393 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 00:31:49.673136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655955    4393 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 00:31:49.673136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655957    4393 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 00:31:49.673136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655960    4393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 00:31:49.673136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655962    4393 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 00:31:49.673136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655964    4393 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 00:31:49.673136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655968    4393 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 00:31:49.673136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655970    4393 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 00:31:49.673136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655972    4393 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 00:31:49.673136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655975    4393 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 00:31:49.673136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655977    4393 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 00:31:49.673136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655980    4393 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 00:31:49.673136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655982    4393 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 00:31:49.673136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655984    4393 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 00:31:49.673136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655987    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 00:31:49.673136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655990    4393 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 00:31:49.673136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655993    4393 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 00:31:49.673136 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655996    4393 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 00:31:49.673609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.655998    4393 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 00:31:49.673609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.656001    4393 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 00:31:49.673609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.656003    4393 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 00:31:49.673609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.656006    4393 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 00:31:49.673609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.656011    4393 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 00:31:49.673609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.656013    4393 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 00:31:49.673609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.656016    4393 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 00:31:49.673609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.656018    4393 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 00:31:49.673609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.656021    4393 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 00:31:49.673609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.656023    4393 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 00:31:49.673609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.656026    4393 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 00:31:49.673609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.656028    4393 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 00:31:49.673609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.656032    4393 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 00:31:49.673609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.656034    4393 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 00:31:49.673609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.656037    4393 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 00:31:49.673609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.656039    4393 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 00:31:49.673609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.656041    4393 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 00:31:49.673609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.656044    4393 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 00:31:49.673609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.656046    4393 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 00:31:49.673609 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.656049    4393 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 00:31:49.674232 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.656051    4393 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 00:31:49.674232 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.656054    4393 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 00:31:49.674232 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.656057    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 00:31:49.674232 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.656059    4393 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 00:31:49.674232 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.656061    4393 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 00:31:49.674232 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:49.656064    4393 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 00:31:49.674232 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.656119    4393 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 00:31:49.674232 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.657128    4393 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 00:31:49.674232 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.657179    4393 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 00:31:49.674232 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.657392    4393 server.go:1014] "Starting client certificate rotation"
Oct 30 00:31:49.674232 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.657531    4393 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 00:31:49.674232 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.657590    4393 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 00:26:09 +0000 UTC" deadline="2028-07-07 10:32:09.75071952 +0000 UTC"
Oct 30 00:31:49.674232 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.657619    4393 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14794h0m20.093102797s"
Oct 30 00:31:49.674232 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.657890    4393 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 00:31:49.674232 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.659491    4393 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 00:31:49.674232 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.661994    4393 log.go:25] "Validated CRI v1 runtime API"
Oct 30 00:31:49.674679 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.663524    4393 log.go:25] "Validated CRI v1 image API"
Oct 30 00:31:49.674679 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.664208    4393 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 00:31:49.674679 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.665710    4393 fs.go:135] Filesystem UUIDs: map[3de2431c-c72f-4541-adb5-2d0f804e005b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ff55ea1e-4b8a-433f-b65f-72d114147a4e:/dev/nvme0n1p3]
Oct 30 00:31:49.674792 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.665729    4393 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/02f558af7f3ca19844c85f4ca8a14e68f33a793e7f1626b2b5140ebbea253e6e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/02f558af7f3ca19844c85f4ca8a14e68f33a793e7f1626b2b5140ebbea253e6e/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/08b687a874a3fd85b6a79f5d3e1ed5360f420760e272fdcc41b9ffd9e44ef64e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/08b687a874a3fd85b6a79f5d3e1ed5360f420760e272fdcc41b9ffd9e44ef64e/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0eb49afe8f3d27e8f679c06d48434c3f565f094fbfde9e391b0fe3ee49e57e9a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0eb49afe8f3d27e8f679c06d48434c3f565f094fbfde9e391b0fe3ee49e57e9a/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/18f770b176e797172e78576db0d907672cf4dc7c54b6d4366fd9eef43baa1f69/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/18f770b176e797172e78576db0d907672cf4dc7c54b6d4366fd9eef43baa1f69/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2891385a68bce7a18baf0301755c0df93db5e2fe2a59bdcb82a33771fb497327/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2891385a68bce7a18baf0301755c0df93db5e2fe2a59bdcb82a33771fb497327/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/30d33f546b4ee2cdbc14239fa2b91763e67dabbe72208cdf284d7622473fdf31/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/30d33f546b4ee2cdbc14239fa2b91763e67dabbe72208cdf284d7622473fdf31/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/62d17e940e9de9de17861e94f0b8198b5892d3baa6cda7fe581b304e78da0ef6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/62d17e940e9de9de17861e94f0b8198b5892d3baa6cda7fe581b304e78da0ef6/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/711a65b4aa0b4b6a781f42961ea6978c5f8abc36884897ab61372cd9bd07c1d1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/711a65b4aa0b4b6a781f42961ea6978c5f8abc36884897ab61372cd9bd07c1d1/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f08e3175a14b08ecca39bbea8f08cfdcbe330df1f4544751cf218dc6e2c11bd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f08e3175a14b08ecca39bbea8f08cfdcbe330df1f4544751cf218dc6e2c11bd/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f882d988379e6b5bccfffdc83df6afc60d627bab3911f201a5e438a435c62a6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f882d988379e6b5bccfffdc83df6afc60d627bab3911f201a5e438a435c62a6/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a00488f5b29d78963136584d8f9efc0625fce53066cf2967bddb4263ed92262/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a00488f5b29d78963136584d8f9efc0625fce53066cf2967bddb4263ed92262/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a0705a5c5f0b4f77899aa6fa69d7e239c6d1e7983fc49afc0d77c8886732f74/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a0705a5c5f0b4f77899aa6fa69d7e239c6d1e7983fc49afc0d77c8886732f74/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b78879c21b93d6a353f9ed0d9d651ec4036b9443832f9c1456ee3d7e0cad7402/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b78879c21b93d6a353f9ed0d9d651ec4036b9443832f9c1456ee3d7e0cad7402/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fd5af83b2e76ace40a2039e61ee3ae28927cf3ab03050ba4dedbb8c116e48408/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fd5af83b2e76ace40a2039e61ee3ae28927cf3ab03050ba4dedbb8c116e48408/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05a614f5-3112-4216-b894-68bf3945458c/volumes/kubernetes.io~projected/kube-api-access-pchtx:{mountpoint:/var/lib/kubelet/pods/05a614f5-3112-4216-b894-68bf3945458c/volumes/kubernetes.io~projected/kube-api-access-pchtx major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835/volumes/kubernetes.io~projected/kube-api-access-bgw94:{mountpoint:/var/lib/kubelet/pods/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835/volumes/kubernetes.io~projected/kube-api-access-bgw94 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30e43c48-92a7-4c68-887c-9aaf7d6d1e1b/volumes/kubernetes.io~projected/kube-api-access-zzcz2:{mountpoint:/var/lib/kubelet/pods/30e43c48-92a7-4c68-887c-9aaf7d6d1e1b/volumes/kubernetes.io~projected/kube-api-access-zzcz2 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b0df054-0749-4417-bb8b-b12b3353d25e/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/3b0df054-0749-4417-bb8b-b12b3353d25e/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b734765-8e7f-4d8e-a74b-bd179adf33ee/volumes/kubernetes.io~projected/kube-api-access-k7rhg:{mountpoint:/var/lib/kubelet/pods/3b734765-8e7f-4d8e-a74b-bd179adf33ee/volumes/kubernetes.io~projected/kube-api-access-k7rhg major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a99683d-3f83-465d-9152-2f422641e087/volumes/kubernetes.io~projected/kube-api-access-28lgt:{mountpoint:/var/lib/kubelet/pods/4a99683d-3f83-465d-9152-2f422641e087/volumes/kubernetes.io~projected/kube-api-access-28lgt major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f7ca654-edeb-4457-9239-8da741b5dafd/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/4f7ca654-edeb-4457-9239-8da741b5dafd/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5687019d-945a-4307-8269-5c0ae6f377ec/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/5687019d-945a-4307-8269-5c0ae6f377ec/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5687019d-945a-4307-8269-5c0ae6f377ec/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/5687019d-945a-4307-8269-5c0ae6f377ec/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5687019d-945a-4307-8269-5c0ae6f377ec/volumes/kubernetes.io~projected/kube-api-access-gqzmf:{mountpoint:/var/lib/kubelet/pods/5687019d-945a-4307-8269-5c0ae6f377ec/volumes/kubernetes.io~projected/kube-api-access-gqzmf major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/626ec6c0-d616-40ae-bf8b-4e8a927c31ac/volumes/kubernetes.io~projected/kube-api-access-g5mw6:{mountpoint:/var/lib/kubelet/pods/626ec6c0-d616-40ae-bf8b-4e8a927c31ac/volumes/kubernetes.io~projected/kube-api-access-g5mw6 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ff169ff-dc1a-4344-b19b-adaabe98f20d/volumes/kubernetes.io~projected/kube-api-access-c4nwk:{mountpoint:/var/lib/kubelet/pods/8ff169ff-dc1a-4344-b19b-adaabe98f20d/volumes/kubernetes.io~projected/kube-api-access-c4nwk major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ff169ff-dc1a-4344-b19b-adaabe98f20d/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/8ff169ff-dc1a-4344-b19b-adaabe98f20d/volumes/kubernetes.io~secret/signing-key major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a20a733f-4947-4a14-9db0-a5be23cb9ccb/volumes/kubernetes.io~projected/kube-api-access-fdxj7:{mountpoint:/var/lib/kubelet/pods/a20a733f-4947-4a14-9db0-a5be23cb9ccb/volumes/kubernetes.io~projected/kube-api-access-fdxj7 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a20a733f-4947-4a14-9db0-a5be23cb9ccb/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/a20a733f-4947-4a14-9db0-a5be23cb9ccb/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3705749-d093-4b49-878d-e4616e81a450/volumes/kubernetes.io~projected/kube-api-access-7xvts:{mountpoint:/var/lib/kubelet/pods/b3705749-d093-4b49-878d-e4616e81a450/volumes/kubernetes.io~projected/kube-api-access-7xvts major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f/volumes/kubernetes.io~projected/kube-api-access-tntlm:{mountpoint:/var/lib/kubelet/pods/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f/volumes/kubernetes.io~projected/kube-api-access-tntlm major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc/volumes/kubernetes.io~projected/kube-api-access-4bgt8:{mountpoint:/var/lib/kubelet/pods/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc/volumes/kubernetes.io~projected/kube-api-access-4bgt8 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e552ba30-5b09-48a7-bffb-6a65a094c2bc/volumes/kubernetes.io~projected/kube-api-access-gpfcq:{mountpoint:/var/lib/kubelet/pods/e552ba30-5b09-48a7-bffb-6a65a094c2bc/volumes/kubernetes.io~projected/kube-api-access-gpfcq major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef9a0a39-051f-45a1-acc9-354a23e5b00b/volumes/kubernetes.io~projected/kube-api-access-wblgx:{mountpoint:/var/lib/kubelet/pods/ef9a0a39-051f-45a1-acc9-354a23e5b00b/volumes/kubernetes.io~projected/kube-api-access-wblgx major:0 minor:62 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/d1b98cb649183c70d43d1cad61ce8df0a108dfa9d2316d2ab30d82f9c933db03/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/ca41df1aff3e2939e4fb02bb66ddb954265bc3bbd9a576271fc22abafecfa9bf/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/299da02e83ca35be9aa75a44e072af75b485970ed5c34bd260c8c6bec85b31a2/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/8bb1ed9d309bd93f16de4845ff0ce3894d5ea7fc4ab07b2d462ea1b06f7fd390/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/7eece907b4566af06e9d3cbf0ce07257ac8de56a2c1b7baeb499231958880889/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/f67dec236494e928bd123d629371e15f4de89fe6f5ac46cafba3906e453f968a/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/b5ebac2bd54750438f4fb3280821707f09f3e9a9d41124d54814944542d62af9/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/2d857db683b6228bae141cf56b197dc89dff5536f05d24a1600ad0ebeebb7f97/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/6217be34110c65a004081c235cc8d0182bfc069dfdc7adcee4a83138375b95d6/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/66de4b698578176e5a1b4acc761232ab9076dc83abac0b3243f5e8a0dd56b64c/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/88b52a34bd62ec77dc348437be96b2530bd22104ec11c48c9a23ed91a9e2690d/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/9ba5331023abefc616f6066bfb5c5bd6a0de37d5824770af82e7fb1538afa2a8/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/8f9d90ae0d872b5df3dacf25c2976a982e178eb35922a610ce473f262f76ac2b/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/e67251679adaf2743cc8654f35dd2d40e6194ce62d8a329eff7f30de2dd8ed43/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-164:{mountpoint:/var/lib/containers/storage/overlay/e1f393bc3445e9b29287924c2d4da30df77a4a85c3aef6ee96c1882b086ec781/merged major:0 minor:164 fsType:overlay blockSize:0} overlay_0-166:{mountpoint:/var/lib/containers/storage/overlay/1a1931a26fa8d7c4babc8a22d42a68f2a760e71f6f2859f846ef0fc3afa5f62b/merged major:0 minor:166 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/3ac575a0033d6bb7b999be55918ce4ffcc1816dbbd9740413d89e9fd30f8d8fb/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/e03cc64779c20d9451989014c86ab59a1155b283426030df30482755b46a3488/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/ecbed85aedd9e8b4521898990f3d5d1d8caca1ee86ad8df6f0b8e6fdb7e176fd/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/c046d1cf648e58020511c94f60f3d8a2e2d3f4058f216cefb4221e9d539028bb/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/51a59e1b51f5d388b3aa38ac9994e0e26adb78e5d18aac65510a5ef8d026d87e/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/54f8f2a8634cd6081cc179e17e18fa4a5c1c9d372243f8380019ffbe8e711a46/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/4d079e91dc1e315b5b11a01ce6b353cd914da1a749b90115b2a673d939cb512e/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/c00f02fbab29ad39b0220a7015ac170c05a132f29a232189882a67db12bb695f/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/2c1037efeb33f87026419d38d950c6f8a223e3009e6de49a2d2ac5aca470e398/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/baeb65a1329441e90630b3d85305d68fed2cce3dc5486ef02540f92274a065e7/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/e8bce4e32c745b32fa105baca88e684ea554776f54c6c5288872bc7ed8476881/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/d2c0ad84ca2e9c856d9cbdcc816f918cea04af3cab4d49eaa4b8ff62af783df2/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/3cde31bd91c410f7151fed3258e274affba7cab82a8c3770c17b58d226cad887/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/b4727dc1efcd00f9e1c6531f336cb83e236af194effe310f1de1d7be371f07f5/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/e08512dcf7f72fe65fe685833b85fd48ab28dd8d355009ae2c1d17230d5c22ac/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/14b3ce738ddbf384ef7d30c88c334a74abf3b9018e11dc42a531f319c3eea80d/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/5e72640ffa310ee84eeff3750529ffa3fbae8a2719ac0759bca9b7753530e815/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/66252953dc3a4dbd1b1c5ccff18e6da0fdb970f157faf02ff998084e42a4480f/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-81:{mountpoint:/var/lib/containers/storage/overlay/0be729997e79a706ed4989c1ac5730f7448795d9354b72156245456dbd4a5f3a/merged major:0 minor:81 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/b46a4b28c88bb41d9ac4fc15e45786d5b25fb51431996ff8edeb767758751b0c/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/9710c61e3fb4833f8f2985dcd5d2a842f14217ae4ba05f9e0b30de87d40bcc1b/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/7ae98a293dd9f792ccb813a3509041cec28daf336acff54c38becf55d0389f5a/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/a14a52bc9e22250e22a24eaf393561b1d87febb2481ed04cd2fe2752816cc970/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 00:31:49.674962 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.672646    4393 manager.go:217] Machine: {Timestamp:2026-10-30 00:31:49.672178114 +0000 UTC m=+0.059854494 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3146515 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c96115e1e07bbe1f2949e21fc3a18 SystemUUID:ec2c9611-5e1e-07bb-e1f2-949e21fc3a18 BootID:faa9a921-8373-404e-8241-d554d5495c82 Filesystems:[{Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b3705749-d093-4b49-878d-e4616e81a450/volumes/kubernetes.io~projected/kube-api-access-7xvts DeviceMajor:0 DeviceMinor:221 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/30e43c48-92a7-4c68-887c-9aaf7d6d1e1b/volumes/kubernetes.io~projected/kube-api-access-zzcz2 DeviceMajor:0 DeviceMinor:65 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/08b687a874a3fd85b6a79f5d3e1ed5360f420760e272fdcc41b9ffd9e44ef64e/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f882d988379e6b5bccfffdc83df6afc60d627bab3911f201a5e438a435c62a6/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5687019d-945a-4307-8269-5c0ae6f377ec/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/30d33f546b4ee2cdbc14239fa2b91763e67dabbe72208cdf284d7622473fdf31/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f08e3175a14b08ecca39bbea8f08cfdcbe330df1f4544751cf218dc6e2c11bd/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f/volumes/kubernetes.io~projected/kube-api-access-tntlm DeviceMajor:0 DeviceMinor:202 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a00488f5b29d78963136584d8f9efc0625fce53066cf2967bddb4263ed92262/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/62d17e940e9de9de17861e94f0b8198b5892d3baa6cda7fe581b304e78da0ef6/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/5687019d-945a-4307-8269-5c0ae6f377ec/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc/volumes/kubernetes.io~projected/kube-api-access-4bgt8 DeviceMajor:0 DeviceMinor:66 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3b734765-8e7f-4d8e-a74b-bd179adf33ee/volumes/kubernetes.io~projected/kube-api-access-k7rhg DeviceMajor:0 DeviceMinor:67 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e552ba30-5b09-48a7-bffb-6a65a094c2bc/volumes/kubernetes.io~projected/kube-api-access-gpfcq DeviceMajor:0 DeviceMinor:60 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4a99683d-3f83-465d-9152-2f422641e087/volumes/kubernetes.io~projected/kube-api-access-28lgt DeviceMajor:0 DeviceMinor:63 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-81 DeviceMajor:0 DeviceMinor:81 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-164 DeviceMajor:0 DeviceMinor:164 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/626ec6c0-d616-40ae-bf8b-4e8a927c31ac/volumes/kubernetes.io~projected/kube-api-access-g5mw6 DeviceMajor:0 DeviceMinor:197 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/05a614f5-3112-4216-b894-68bf3945458c/volumes/kubernetes.io~projected/kube-api-access-pchtx DeviceMajor:0 DeviceMinor:59 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/711a65b4aa0b4b6a781f42961ea6978c5f8abc36884897ab61372cd9bd07c1d1/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/02f558af7f3ca19844c85f4ca8a14e68f33a793e7f1626b2b5140ebbea253e6e/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a20a733f-4947-4a14-9db0-a5be23cb9ccb/volumes/kubernetes.io~projected/kube-api-access-fdxj7 DeviceMajor:0 DeviceMinor:201 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b78879c21b93d6a353f9ed0d9d651ec4036b9443832f9c1456ee3d7e0cad7402/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8ff169ff-dc1a-4344-b19b-adaabe98f20d/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:207 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4f7ca654-edeb-4457-9239-8da741b5dafd/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2891385a68bce7a18baf0301755c0df93db5e2fe2a59bdcb82a33771fb497327/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-166 DeviceMajor:0 DeviceMinor:166 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8ff169ff-dc1a-4344-b19b-adaabe98f20d/volumes/kubernetes.io~projected/kube-api-access-c4nwk DeviceMajor:0 DeviceMinor:208 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5687019d-945a-4307-8269-5c0ae6f377ec/volumes/kubernetes.io~projected/kube-api-access-gqzmf DeviceMajor:0 DeviceMinor:61 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/ef9a0a39-051f-45a1-acc9-354a23e5b00b/volumes/kubernetes.io~projected/kube-api-access-wblgx DeviceMajor:0 DeviceMinor:62 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0eb49afe8f3d27e8f679c06d48434c3f565f094fbfde9e391b0fe3ee49e57e9a/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/3b0df054-0749-4417-bb8b-b12b3353d25e/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:226 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/18f770b176e797172e78576db0d907672cf4dc7c54b6d4366fd9eef43baa1f69/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835/volumes/kubernetes.io~projected/kube-api-access-bgw94 DeviceMajor:0 DeviceMinor:64 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a20a733f-4947-4a14-9db0-a5be23cb9ccb/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a0705a5c5f0b4f77899aa6fa69d7e239c6d1e7983fc49afc0d77c8886732f74/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fd5af83b2e76ace40a2039e61ee3ae28927cf3ab03050ba4dedbb8c116e48408/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0eb49afe8f3d27e MacAddress:de:18:ee:64:9a:57 Speed:10000 Mtu:8901} {Name:18f770b176e7971 MacAddress:1a:16:02:14:57:7a Speed:10000 Mtu:8901} {Name:9a0705a5c5f0b4f MacAddress:d2:33:7b:ec:de:06 Speed:10000 Mtu:8901} {Name:b78879c21b93d6a MacAddress:52:d5:84:7f:fc:42 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:dd:6f:ac:a5 Speed:0 Mtu:9001} {Name:br-int MacAddress:7e:8f:3c:7b:f7:78 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:dd:6f:ac:a5 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:9a:b6:30:71:fb:09 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:5a:c1:57:0f:ad:1b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 00:31:49.674962 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.672917    4393 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 00:31:49.674962 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.672964    4393 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 00:31:49.674962 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.673268    4393 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 00:31:49.674962 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.673289    4393 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-56.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 00:31:49.674962 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.673433    4393 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 00:31:49.674962 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.673441    4393 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 00:31:49.674962 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.673455    4393 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 00:31:49.674962 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.673474    4393 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 00:31:49.674962 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.673578    4393 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 00:31:49.674962 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.673745    4393 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 00:31:49.674962 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.673873    4393 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 00:31:49.674962 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.673909    4393 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 00:31:49.674962 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.673922    4393 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 00:31:49.674962 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.673932    4393 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 00:31:49.674962 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.673942    4393 kubelet.go:405] "Adding apiserver pod source"
Oct 30 00:31:49.674962 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.673950    4393 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 00:31:49.674962 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.674563    4393 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 00:31:49.674962 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.674572    4393 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 00:31:49.675964 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.675062    4393 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 00:31:49.678548 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.678519    4393 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 00:31:49.678548 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.678544    4393 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 00:31:49.678686 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.678675    4393 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 00:31:49.678961 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.678949    4393 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 00:31:49.679003 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.678980    4393 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 00:31:49.679080 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.679072    4393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 00:31:49.679115 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.679090    4393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 00:31:49.679115 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.679099    4393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 00:31:49.679115 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.679107    4393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 00:31:49.679115 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.679113    4393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 00:31:49.679212 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.679119    4393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 00:31:49.679212 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.679125    4393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 00:31:49.679212 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.679131    4393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 00:31:49.679212 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.679138    4393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 00:31:49.679212 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.679144    4393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 00:31:49.679212 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.679159    4393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 00:31:49.679212 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.679170    4393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 00:31:49.679212 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.679198    4393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 00:31:49.679212 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.679209    4393 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 00:31:49.682872 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.682762    4393 server.go:1263] "Started kubelet"
Oct 30 00:31:49.682960 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.682869    4393 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 00:31:49.683157 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.683104    4393 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 00:31:49.683216 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.683182    4393 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 00:31:49.683386 ip-10-0-135-56 systemd[1]: Started Kubernetes Kubelet.
Oct 30 00:31:49.683659 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.683639    4393 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 00:31:49.685155 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.684154    4393 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 00:31:49.685592 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.685324    4393 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 00:31:49.685592 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.685376    4393 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 00:31:49.685592 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:31:49.685441    4393 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-56.ec2.internal\" not found"
Oct 30 00:31:49.685592 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.685431    4393 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 00:26:11 +0000 UTC" deadline="2028-08-07 11:09:37.037977538 +0000 UTC"
Oct 30 00:31:49.685592 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.685468    4393 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15538h37m47.352513515s"
Oct 30 00:31:49.690295 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.690254    4393 factory.go:55] Registering systemd factory
Oct 30 00:31:49.690295 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.690282    4393 factory.go:223] Registration of the systemd container factory successfully
Oct 30 00:31:49.690905 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.690865    4393 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 00:31:49.691020 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.691001    4393 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 00:31:49.691117 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.690937    4393 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 00:31:49.691511 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.691485    4393 factory.go:153] Registering CRI-O factory
Oct 30 00:31:49.691511 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.691509    4393 factory.go:223] Registration of the crio container factory successfully
Oct 30 00:31:49.691682 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.691565    4393 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 00:31:49.691682 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.691587    4393 factory.go:103] Registering Raw factory
Oct 30 00:31:49.691682 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.691633    4393 manager.go:1199] Started watching for new ooms in manager
Oct 30 00:31:49.692096 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.692080    4393 manager.go:319] Starting recovery of all containers
Oct 30 00:31:49.693019 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.692959    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05a614f5-3112-4216-b894-68bf3945458c" volumeName="kubernetes.io/projected/05a614f5-3112-4216-b894-68bf3945458c-kube-api-access-pchtx" seLinuxMountContext=""
Oct 30 00:31:49.693019 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693011    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a99683d-3f83-465d-9152-2f422641e087" volumeName="kubernetes.io/configmap/4a99683d-3f83-465d-9152-2f422641e087-iptables-alerter-script" seLinuxMountContext=""
Oct 30 00:31:49.693199 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693038    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef9a0a39-051f-45a1-acc9-354a23e5b00b" volumeName="kubernetes.io/configmap/ef9a0a39-051f-45a1-acc9-354a23e5b00b-cni-binary-copy" seLinuxMountContext=""
Oct 30 00:31:49.693199 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693056    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835" volumeName="kubernetes.io/configmap/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-ovnkube-config" seLinuxMountContext=""
Oct 30 00:31:49.693199 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693085    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a99683d-3f83-465d-9152-2f422641e087" volumeName="kubernetes.io/projected/4a99683d-3f83-465d-9152-2f422641e087-kube-api-access-28lgt" seLinuxMountContext=""
Oct 30 00:31:49.693199 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693107    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5687019d-945a-4307-8269-5c0ae6f377ec" volumeName="kubernetes.io/empty-dir/5687019d-945a-4307-8269-5c0ae6f377ec-tmp" seLinuxMountContext=""
Oct 30 00:31:49.693199 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693123    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5687019d-945a-4307-8269-5c0ae6f377ec" volumeName="kubernetes.io/projected/5687019d-945a-4307-8269-5c0ae6f377ec-kube-api-access-gqzmf" seLinuxMountContext=""
Oct 30 00:31:49.693199 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693136    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3705749-d093-4b49-878d-e4616e81a450" volumeName="kubernetes.io/projected/b3705749-d093-4b49-878d-e4616e81a450-kube-api-access-7xvts" seLinuxMountContext=""
Oct 30 00:31:49.693199 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693164    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f" volumeName="kubernetes.io/configmap/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-config-volume" seLinuxMountContext=""
Oct 30 00:31:49.693199 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693178    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef9a0a39-051f-45a1-acc9-354a23e5b00b" volumeName="kubernetes.io/configmap/ef9a0a39-051f-45a1-acc9-354a23e5b00b-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 00:31:49.693199 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693193    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835" volumeName="kubernetes.io/projected/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-kube-api-access-bgw94" seLinuxMountContext=""
Oct 30 00:31:49.693628 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693216    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30e43c48-92a7-4c68-887c-9aaf7d6d1e1b" volumeName="kubernetes.io/projected/30e43c48-92a7-4c68-887c-9aaf7d6d1e1b-kube-api-access-zzcz2" seLinuxMountContext=""
Oct 30 00:31:49.693628 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693230    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f7ca654-edeb-4457-9239-8da741b5dafd" volumeName="kubernetes.io/configmap/4f7ca654-edeb-4457-9239-8da741b5dafd-konnectivity-ca" seLinuxMountContext=""
Oct 30 00:31:49.693628 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693244    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f7ca654-edeb-4457-9239-8da741b5dafd" volumeName="kubernetes.io/secret/4f7ca654-edeb-4457-9239-8da741b5dafd-agent-certs" seLinuxMountContext=""
Oct 30 00:31:49.693628 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693270    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="626ec6c0-d616-40ae-bf8b-4e8a927c31ac" volumeName="kubernetes.io/empty-dir/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-data-volume" seLinuxMountContext=""
Oct 30 00:31:49.693628 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693320    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ff169ff-dc1a-4344-b19b-adaabe98f20d" volumeName="kubernetes.io/configmap/8ff169ff-dc1a-4344-b19b-adaabe98f20d-signing-cabundle" seLinuxMountContext=""
Oct 30 00:31:49.693628 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693333    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f" volumeName="kubernetes.io/empty-dir/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-tmp-dir" seLinuxMountContext=""
Oct 30 00:31:49.693628 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693356    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef9a0a39-051f-45a1-acc9-354a23e5b00b" volumeName="kubernetes.io/configmap/ef9a0a39-051f-45a1-acc9-354a23e5b00b-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 00:31:49.693628 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693371    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05a614f5-3112-4216-b894-68bf3945458c" volumeName="kubernetes.io/empty-dir/05a614f5-3112-4216-b894-68bf3945458c-tmp-dir" seLinuxMountContext=""
Oct 30 00:31:49.693628 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693383    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835" volumeName="kubernetes.io/configmap/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-env-overrides" seLinuxMountContext=""
Oct 30 00:31:49.693628 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693417    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835" volumeName="kubernetes.io/secret/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 00:31:49.693628 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693433    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b0df054-0749-4417-bb8b-b12b3353d25e" volumeName="kubernetes.io/secret/3b0df054-0749-4417-bb8b-b12b3353d25e-original-pull-secret" seLinuxMountContext=""
Oct 30 00:31:49.693628 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693446    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f" volumeName="kubernetes.io/projected/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-kube-api-access-tntlm" seLinuxMountContext=""
Oct 30 00:31:49.693628 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693479    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef9a0a39-051f-45a1-acc9-354a23e5b00b" volumeName="kubernetes.io/projected/ef9a0a39-051f-45a1-acc9-354a23e5b00b-kube-api-access-wblgx" seLinuxMountContext=""
Oct 30 00:31:49.693628 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693511    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b734765-8e7f-4d8e-a74b-bd179adf33ee" volumeName="kubernetes.io/projected/3b734765-8e7f-4d8e-a74b-bd179adf33ee-kube-api-access-k7rhg" seLinuxMountContext=""
Oct 30 00:31:49.693628 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693523    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="626ec6c0-d616-40ae-bf8b-4e8a927c31ac" volumeName="kubernetes.io/projected/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-kube-api-access-g5mw6" seLinuxMountContext=""
Oct 30 00:31:49.693628 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693549    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc" volumeName="kubernetes.io/projected/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-kube-api-access-4bgt8" seLinuxMountContext=""
Oct 30 00:31:49.693628 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693588    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e552ba30-5b09-48a7-bffb-6a65a094c2bc" volumeName="kubernetes.io/projected/e552ba30-5b09-48a7-bffb-6a65a094c2bc-kube-api-access-gpfcq" seLinuxMountContext=""
Oct 30 00:31:49.694563 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693656    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5687019d-945a-4307-8269-5c0ae6f377ec" volumeName="kubernetes.io/empty-dir/5687019d-945a-4307-8269-5c0ae6f377ec-etc-tuned" seLinuxMountContext=""
Oct 30 00:31:49.694563 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693705    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ff169ff-dc1a-4344-b19b-adaabe98f20d" volumeName="kubernetes.io/secret/8ff169ff-dc1a-4344-b19b-adaabe98f20d-signing-key" seLinuxMountContext=""
Oct 30 00:31:49.694563 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693717    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a20a733f-4947-4a14-9db0-a5be23cb9ccb" volumeName="kubernetes.io/projected/a20a733f-4947-4a14-9db0-a5be23cb9ccb-kube-api-access-fdxj7" seLinuxMountContext=""
Oct 30 00:31:49.694563 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693744    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30e43c48-92a7-4c68-887c-9aaf7d6d1e1b" volumeName="kubernetes.io/configmap/30e43c48-92a7-4c68-887c-9aaf7d6d1e1b-serviceca" seLinuxMountContext=""
Oct 30 00:31:49.694563 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693815    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a20a733f-4947-4a14-9db0-a5be23cb9ccb" volumeName="kubernetes.io/secret/a20a733f-4947-4a14-9db0-a5be23cb9ccb-cert" seLinuxMountContext=""
Oct 30 00:31:49.694563 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693852    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835" volumeName="kubernetes.io/configmap/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 00:31:49.694563 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693865    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b734765-8e7f-4d8e-a74b-bd179adf33ee" volumeName="kubernetes.io/configmap/3b734765-8e7f-4d8e-a74b-bd179adf33ee-multus-daemon-config" seLinuxMountContext=""
Oct 30 00:31:49.694563 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693928    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b734765-8e7f-4d8e-a74b-bd179adf33ee" volumeName="kubernetes.io/configmap/3b734765-8e7f-4d8e-a74b-bd179adf33ee-cni-binary-copy" seLinuxMountContext=""
Oct 30 00:31:49.694563 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693968    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="626ec6c0-d616-40ae-bf8b-4e8a927c31ac" volumeName="kubernetes.io/configmap/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 00:31:49.694563 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693983    4393 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ff169ff-dc1a-4344-b19b-adaabe98f20d" volumeName="kubernetes.io/projected/8ff169ff-dc1a-4344-b19b-adaabe98f20d-kube-api-access-c4nwk" seLinuxMountContext=""
Oct 30 00:31:49.694563 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.693994    4393 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 00:31:49.694563 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.694001    4393 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 00:31:49.714899 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.714841    4393 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 00:31:49.716308 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.716272    4393 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 00:31:49.716308 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.716297    4393 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 00:31:49.716308 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.716316    4393 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 00:31:49.716500 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:31:49.716353    4393 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 00:31:49.736070 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.736043    4393 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 00:31:49.740935 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.740909    4393 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:31:49.741133 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.741111    4393 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:31:49.745798 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.745704    4393 generic.go:356] "Generic (PLEG): container finished" podID="ef9a0a39-051f-45a1-acc9-354a23e5b00b" containerID="6b8776286e3ae617ffd413c0780fb003bbf21e2d57d4914514298b6f408ebc32" exitCode=0
Oct 30 00:31:49.745798 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.745753    4393 generic.go:356] "Generic (PLEG): container finished" podID="ef9a0a39-051f-45a1-acc9-354a23e5b00b" containerID="2740e9825f42fce105efafe77efdce4b6a01bcde5ed71d79fa66c9e683f1120b" exitCode=0
Oct 30 00:31:49.745798 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.745766    4393 generic.go:356] "Generic (PLEG): container finished" podID="ef9a0a39-051f-45a1-acc9-354a23e5b00b" containerID="a115a4e49fdd94fd1c4243efca7324e804b57a9d62324c2563c678d18950a5c7" exitCode=0
Oct 30 00:31:49.745798 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.745794    4393 generic.go:356] "Generic (PLEG): container finished" podID="ef9a0a39-051f-45a1-acc9-354a23e5b00b" containerID="eb660c3e5e3926db0ed2cca4dbce6808d91745769650428aa70cf4b5b4654c8d" exitCode=0
Oct 30 00:31:49.745798 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.745806    4393 generic.go:356] "Generic (PLEG): container finished" podID="ef9a0a39-051f-45a1-acc9-354a23e5b00b" containerID="13dc4a5cea096cfe80d09b3d12f5bbe3bb607811d3eecd9c65c5dd4fe5f42a78" exitCode=0
Oct 30 00:31:49.746186 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.745814    4393 generic.go:356] "Generic (PLEG): container finished" podID="ef9a0a39-051f-45a1-acc9-354a23e5b00b" containerID="8fb55fece667206aca4113a3508bf1902142200d36f6997471005c9313e00ad0" exitCode=0
Oct 30 00:31:49.750232 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.750201    4393 generic.go:356] "Generic (PLEG): container finished" podID="f3ebcfeede45130240ea614a61281a18" containerID="37ac9b4631a778e3a0b9dfa9c14f01a951ec5805b9833c8d20097088bf7fba89" exitCode=0
Oct 30 00:31:49.753270 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.753235    4393 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:31:49.753391 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.753330    4393 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 00:31:49.762279 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.762255    4393 manager.go:324] Recovery completed
Oct 30 00:31:49.779598 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.779522    4393 cpu_manager.go:226] "Starting" policy="none"
Oct 30 00:31:49.779598 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.779538    4393 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 00:31:49.779598 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.779559    4393 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 00:31:49.779850 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.779697    4393 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 00:31:49.779850 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.779708    4393 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 00:31:49.779850 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.779725    4393 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 00:31:49.779850 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.779731    4393 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 00:31:49.779850 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.779737    4393 policy_none.go:50] "Start"
Oct 30 00:31:49.779850 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.779753    4393 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 00:31:49.779850 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.779763    4393 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 00:31:49.780073 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.779907    4393 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 00:31:49.780073 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.779914    4393 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 00:31:49.780073 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.779920    4393 policy_none.go:44] "Start"
Oct 30 00:31:49.784088 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.784056    4393 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 00:31:49.786752 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.786732    4393 manager.go:341] "Starting Device Plugin manager"
Oct 30 00:31:49.787257 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:31:49.786762    4393 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 00:31:49.787299 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.787261    4393 server.go:87] "Starting device plugin registration server"
Oct 30 00:31:49.787551 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.787542    4393 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 00:31:49.787580 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.787554    4393 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 00:31:49.787687 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.787668    4393 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 00:31:49.787861 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.787808    4393 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 00:31:49.787975 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.787922    4393 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 00:31:49.787975 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.787927    4393 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 00:31:49.788056 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.787992    4393 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T00:31:49.787856024Z","UUID":"c5238c91-5d72-4814-acf0-629ec62f7792","Handler":null,"Name":"","Endpoint":""}
Oct 30 00:31:49.789108 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:31:49.789081    4393 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 00:31:49.789324 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.789297    4393 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 00:31:49.789400 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.789347    4393 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 00:31:49.816956 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.816901    4393 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-56.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal"]
Oct 30 00:31:49.817412 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.817371    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-56.ec2.internal" event={"ID":"21539217511a9b702eb77d59ab0b339d","Type":"ContainerStarted","Data":"db9b07011d2f050f681fa96d04710e99dfd894cfae442af0b42780ee1d573d36"}
Oct 30 00:31:49.817412 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.817410    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-56.ec2.internal" event={"ID":"21539217511a9b702eb77d59ab0b339d","Type":"ContainerStarted","Data":"62d17e940e9de9de17861e94f0b8198b5892d3baa6cda7fe581b304e78da0ef6"}
Oct 30 00:31:49.817555 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.817455    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal" event={"ID":"f3ebcfeede45130240ea614a61281a18","Type":"ContainerStarted","Data":"3d15129431a215893a9646f835eb4259905f2df4159fe7794fc9182186624040"}
Oct 30 00:31:49.817555 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.817473    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal" event={"ID":"f3ebcfeede45130240ea614a61281a18","Type":"ContainerDied","Data":"37ac9b4631a778e3a0b9dfa9c14f01a951ec5805b9833c8d20097088bf7fba89"}
Oct 30 00:31:49.817555 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.817486    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal" event={"ID":"f3ebcfeede45130240ea614a61281a18","Type":"ContainerStarted","Data":"fd5af83b2e76ace40a2039e61ee3ae28927cf3ab03050ba4dedbb8c116e48408"}
Oct 30 00:31:49.817555 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.817524    4393 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-56.ec2.internal"
Oct 30 00:31:49.817692 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.817590    4393 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal"
Oct 30 00:31:49.840024 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.839994    4393 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:31:49.840179 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:31:49.840050    4393 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-56.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-56.ec2.internal"
Oct 30 00:31:49.840502 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.840491    4393 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:31:49.840540 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:31:49.840514    4393 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal"
Oct 30 00:31:49.888662 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.888626    4393 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 00:31:49.890262 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.890238    4393 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-56.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 00:31:49.890402 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.890279    4393 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-56.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 00:31:49.890402 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.890295    4393 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-56.ec2.internal" event="NodeHasSufficientPID"
Oct 30 00:31:49.890402 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.890345    4393 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-56.ec2.internal"
Oct 30 00:31:49.923367 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.923341    4393 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-135-56.ec2.internal"
Oct 30 00:31:49.923431 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.923423    4393 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-56.ec2.internal"
Oct 30 00:31:49.937268 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.937239    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3ebcfeede45130240ea614a61281a18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal\" (UID: \"f3ebcfeede45130240ea614a61281a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal"
Oct 30 00:31:49.937413 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.937275    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/21539217511a9b702eb77d59ab0b339d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-56.ec2.internal\" (UID: \"21539217511a9b702eb77d59ab0b339d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-56.ec2.internal"
Oct 30 00:31:49.937413 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:49.937293    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f3ebcfeede45130240ea614a61281a18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal\" (UID: \"f3ebcfeede45130240ea614a61281a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal"
Oct 30 00:31:50.038132 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.038029    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3ebcfeede45130240ea614a61281a18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal\" (UID: \"f3ebcfeede45130240ea614a61281a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal"
Oct 30 00:31:50.038132 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.038071    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/21539217511a9b702eb77d59ab0b339d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-56.ec2.internal\" (UID: \"21539217511a9b702eb77d59ab0b339d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-56.ec2.internal"
Oct 30 00:31:50.038132 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.038093    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f3ebcfeede45130240ea614a61281a18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal\" (UID: \"f3ebcfeede45130240ea614a61281a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal"
Oct 30 00:31:50.038334 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.038151    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f3ebcfeede45130240ea614a61281a18-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal\" (UID: \"f3ebcfeede45130240ea614a61281a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal"
Oct 30 00:31:50.038334 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.038153    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/21539217511a9b702eb77d59ab0b339d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-56.ec2.internal\" (UID: \"21539217511a9b702eb77d59ab0b339d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-56.ec2.internal"
Oct 30 00:31:50.038334 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.038155    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3ebcfeede45130240ea614a61281a18-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal\" (UID: \"f3ebcfeede45130240ea614a61281a18\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal"
Oct 30 00:31:50.674721 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.674677    4393 apiserver.go:52] "Watching apiserver"
Oct 30 00:31:50.713807 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.713752    4393 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 00:31:50.714249 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.714228    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-jldmt","openshift-service-ca/service-ca-5fffc97bdf-mln7q","openshift-cluster-node-tuning-operator/tuned-lnnlc","openshift-dns/dns-default-9ml2g","openshift-insights/insights-runtime-extractor-jbmfs","openshift-multus/multus-r58sp","openshift-ovn-kubernetes/ovnkube-node-pxfhp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz","openshift-image-registry/node-ca-9hslh","kube-system/konnectivity-agent-6wzd9","kube-system/global-pull-secret-syncer-f27vk","openshift-network-operator/iptables-alerter-s72l8","openshift-multus/multus-additional-cni-plugins-csc84","kube-system/kube-apiserver-proxy-ip-10-0-135-56.ec2.internal","openshift-dns/node-resolver-fqtzs","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal","openshift-ingress-canary/ingress-canary-jb5sp","openshift-network-diagnostics/network-check-target-wpvjg"]
Oct 30 00:31:50.720791 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.720750    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jldmt"
Oct 30 00:31:50.721253 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.721230    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:50.721771 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.721755    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9ml2g"
Oct 30 00:31:50.729450 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.729425    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-z22kq\""
Oct 30 00:31:50.729574 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.729527    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 00:31:50.729651 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.729585    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-rw2kd\""
Oct 30 00:31:50.729707 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.729652    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-ttql7\""
Oct 30 00:31:50.729707 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.729671    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:31:50.729707 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.729673    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 00:31:50.729829 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.729726    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 00:31:50.729829 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.729763    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 00:31:50.729829 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.729761    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 00:31:50.729829 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.729813    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 00:31:50.738773 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.738753    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mt5kv\""
Oct 30 00:31:50.739741 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.739721    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 00:31:50.739741 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.739737    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 00:31:50.739871 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.739760    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:31:50.740008 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.739995    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 00:31:50.740008 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.740006    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 00:31:50.740243 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.740231    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 00:31:50.740578 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.740563    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 00:31:50.741249 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.741237    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6rb57\""
Oct 30 00:31:50.741600 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.741582    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6gpzw\""
Oct 30 00:31:50.741684 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.741599    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 00:31:50.741684 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.741604    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 00:31:50.741684 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.741618    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 00:31:50.741684 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.741647    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-nrfgz\""
Oct 30 00:31:50.741888 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.741730    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 00:31:50.741888 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.741736    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 00:31:50.741888 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.741757    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8g889\""
Oct 30 00:31:50.741888 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.741814    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 00:31:50.741888 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.741834    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 00:31:50.741888 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.741858    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 00:31:50.742119 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.741877    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 00:31:50.742119 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.741877    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 00:31:50.742119 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.741892    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 00:31:50.742119 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.741957    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-h6pr4\""
Oct 30 00:31:50.742119 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.742005    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 00:31:50.742119 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.742007    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 00:31:50.742119 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.742045    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 00:31:50.742119 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.742075    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 00:31:50.742119 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.742122    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fch7k\""
Oct 30 00:31:50.742473 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.742149    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 00:31:50.742473 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.742342    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 00:31:50.742552 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.742525    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 00:31:50.742552 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.742545    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 00:31:50.742696 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.742682    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 00:31:50.742736 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.742684    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 00:31:50.742736 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.742703    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 00:31:50.742825 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.742737    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-qfggf\""
Oct 30 00:31:50.742825 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.742758    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 00:31:50.742825 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.742811    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 00:31:50.742825 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.742822    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 00:31:50.743098 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.742856    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 00:31:50.743098 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.742921    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 00:31:50.743098 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.742931    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8njw4\""
Oct 30 00:31:50.743098 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.742934    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ngd4v\""
Oct 30 00:31:50.743098 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.742993    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 00:31:50.743098 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.743072    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 00:31:50.743098 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.743086    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xlz4l\""
Oct 30 00:31:50.743402 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.743210    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 00:31:50.743402 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.743373    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-d7h4r\""
Oct 30 00:31:50.743825 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.743797    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-xrsjq\""
Oct 30 00:31:50.758072 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.758040    4393 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal"
Oct 30 00:31:50.758262 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.758144    4393 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-56.ec2.internal"
Oct 30 00:31:50.781467 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.781434    4393 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:31:50.781623 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:31:50.781496    4393 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-56.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-56.ec2.internal"
Oct 30 00:31:50.781895 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.781880    4393 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 00:31:50.781935 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:31:50.781917    4393 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-56.ec2.internal"
Oct 30 00:31:50.793233 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.793211    4393 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 00:31:50.841236 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.841197    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ef9a0a39-051f-45a1-acc9-354a23e5b00b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:50.841236 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.841234    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqzmf\" (UniqueName: \"kubernetes.io/projected/5687019d-945a-4307-8269-5c0ae6f377ec-kube-api-access-gqzmf\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.841474 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.841255    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8ff169ff-dc1a-4344-b19b-adaabe98f20d-signing-cabundle\") pod \"service-ca-5fffc97bdf-mln7q\" (UID: \"8ff169ff-dc1a-4344-b19b-adaabe98f20d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mln7q"
Oct 30 00:31:50.841474 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.841278    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28lgt\" (UniqueName: \"kubernetes.io/projected/4a99683d-3f83-465d-9152-2f422641e087-kube-api-access-28lgt\") pod \"iptables-alerter-s72l8\" (UID: \"4a99683d-3f83-465d-9152-2f422641e087\") " pod="openshift-network-operator/iptables-alerter-s72l8"
Oct 30 00:31:50.841474 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.841301    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-etc-sysconfig\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.841474 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.841319    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3b0df054-0749-4417-bb8b-b12b3353d25e-dbus\") pod \"global-pull-secret-syncer-f27vk\" (UID: \"3b0df054-0749-4417-bb8b-b12b3353d25e\") " pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:50.841474 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.841440    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-run-netns\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.841715 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.841487    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-ovnkube-script-lib\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.841715 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.841529    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8ff169ff-dc1a-4344-b19b-adaabe98f20d-signing-cabundle\") pod \"service-ca-5fffc97bdf-mln7q\" (UID: \"8ff169ff-dc1a-4344-b19b-adaabe98f20d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mln7q"
Oct 30 00:31:50.841715 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.841556    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-registration-dir\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:50.841715 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.841601    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-etc-selinux\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:50.841715 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.841647    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-metrics-tls\") pod \"dns-default-9ml2g\" (UID: \"b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f\") " pod="openshift-dns/dns-default-9ml2g"
Oct 30 00:31:50.841715 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.841651    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ef9a0a39-051f-45a1-acc9-354a23e5b00b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:50.841715 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.841702    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/30e43c48-92a7-4c68-887c-9aaf7d6d1e1b-serviceca\") pod \"node-ca-9hslh\" (UID: \"30e43c48-92a7-4c68-887c-9aaf7d6d1e1b\") " pod="openshift-image-registry/node-ca-9hslh"
Oct 30 00:31:50.842176 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.841739    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-run-openvswitch\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.842176 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.841771    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-run-ovn\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.842176 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.841839    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ef9a0a39-051f-45a1-acc9-354a23e5b00b-system-cni-dir\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:50.842176 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.841912    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wblgx\" (UniqueName: \"kubernetes.io/projected/ef9a0a39-051f-45a1-acc9-354a23e5b00b-kube-api-access-wblgx\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:50.842176 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.841939    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-ovnkube-script-lib\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.842176 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.841958    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05a614f5-3112-4216-b894-68bf3945458c-tmp-dir\") pod \"node-resolver-fqtzs\" (UID: \"05a614f5-3112-4216-b894-68bf3945458c\") " pod="openshift-dns/node-resolver-fqtzs"
Oct 30 00:31:50.842176 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.842076    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:50.842176 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.842120    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4nwk\" (UniqueName: \"kubernetes.io/projected/8ff169ff-dc1a-4344-b19b-adaabe98f20d-kube-api-access-c4nwk\") pod \"service-ca-5fffc97bdf-mln7q\" (UID: \"8ff169ff-dc1a-4344-b19b-adaabe98f20d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mln7q"
Oct 30 00:31:50.842176 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.842160    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-system-cni-dir\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.842176 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.842123    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05a614f5-3112-4216-b894-68bf3945458c-tmp-dir\") pod \"node-resolver-fqtzs\" (UID: \"05a614f5-3112-4216-b894-68bf3945458c\") " pod="openshift-dns/node-resolver-fqtzs"
Oct 30 00:31:50.842642 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.842185    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a20a733f-4947-4a14-9db0-a5be23cb9ccb-cert\") pod \"ingress-canary-jb5sp\" (UID: \"a20a733f-4947-4a14-9db0-a5be23cb9ccb\") " pod="openshift-ingress-canary/ingress-canary-jb5sp"
Oct 30 00:31:50.842642 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.842213    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-node-log\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.842642 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.842238    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-ovn-node-metrics-cert\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.842642 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.842259    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4f7ca654-edeb-4457-9239-8da741b5dafd-konnectivity-ca\") pod \"konnectivity-agent-6wzd9\" (UID: \"4f7ca654-edeb-4457-9239-8da741b5dafd\") " pod="kube-system/konnectivity-agent-6wzd9"
Oct 30 00:31:50.842642 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.842274    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-etc-systemd\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.842642 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.842338    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5687019d-945a-4307-8269-5c0ae6f377ec-etc-tuned\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.842642 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.842365    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pchtx\" (UniqueName: \"kubernetes.io/projected/05a614f5-3112-4216-b894-68bf3945458c-kube-api-access-pchtx\") pod \"node-resolver-fqtzs\" (UID: \"05a614f5-3112-4216-b894-68bf3945458c\") " pod="openshift-dns/node-resolver-fqtzs"
Oct 30 00:31:50.842642 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.842420    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a20a733f-4947-4a14-9db0-a5be23cb9ccb-cert\") pod \"ingress-canary-jb5sp\" (UID: \"a20a733f-4947-4a14-9db0-a5be23cb9ccb\") " pod="openshift-ingress-canary/ingress-canary-jb5sp"
Oct 30 00:31:50.842642 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.842451    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4f7ca654-edeb-4457-9239-8da741b5dafd-konnectivity-ca\") pod \"konnectivity-agent-6wzd9\" (UID: \"4f7ca654-edeb-4457-9239-8da741b5dafd\") " pod="kube-system/konnectivity-agent-6wzd9"
Oct 30 00:31:50.842642 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.842429    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-host-var-lib-cni-bin\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.842642 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.842466    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5687019d-945a-4307-8269-5c0ae6f377ec-etc-tuned\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.842642 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.842497    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3b0df054-0749-4417-bb8b-b12b3353d25e-kubelet-config\") pod \"global-pull-secret-syncer-f27vk\" (UID: \"3b0df054-0749-4417-bb8b-b12b3353d25e\") " pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:50.842642 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.842545    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-slash\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.842642 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.842580    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-log-socket\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.842642 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.842581    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/30e43c48-92a7-4c68-887c-9aaf7d6d1e1b-serviceca\") pod \"node-ca-9hslh\" (UID: \"30e43c48-92a7-4c68-887c-9aaf7d6d1e1b\") " pod="openshift-image-registry/node-ca-9hslh"
Oct 30 00:31:50.842642 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.842642    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-run-ovn-kubernetes\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.843255 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.842677    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bgt8\" (UniqueName: \"kubernetes.io/projected/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-kube-api-access-4bgt8\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:50.843255 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.842705    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-data-volume\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:50.843255 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.842731    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8ff169ff-dc1a-4344-b19b-adaabe98f20d-signing-key\") pod \"service-ca-5fffc97bdf-mln7q\" (UID: \"8ff169ff-dc1a-4344-b19b-adaabe98f20d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mln7q"
Oct 30 00:31:50.843255 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.842740    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-ovn-node-metrics-cert\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.843255 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.842758    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3b734765-8e7f-4d8e-a74b-bd179adf33ee-multus-daemon-config\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.843255 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.842806    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-kubelet\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.843255 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.842843    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ef9a0a39-051f-45a1-acc9-354a23e5b00b-cni-binary-copy\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:50.843255 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.842829    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-data-volume\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:50.843255 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.842882    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-multus-socket-dir-parent\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.843255 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.842909    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-hostroot\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.843255 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.842966    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-multus-conf-dir\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.843255 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.842985    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzcz2\" (UniqueName: \"kubernetes.io/projected/30e43c48-92a7-4c68-887c-9aaf7d6d1e1b-kube-api-access-zzcz2\") pod \"node-ca-9hslh\" (UID: \"30e43c48-92a7-4c68-887c-9aaf7d6d1e1b\") " pod="openshift-image-registry/node-ca-9hslh"
Oct 30 00:31:50.843255 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.843010    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8ff169ff-dc1a-4344-b19b-adaabe98f20d-signing-key\") pod \"service-ca-5fffc97bdf-mln7q\" (UID: \"8ff169ff-dc1a-4344-b19b-adaabe98f20d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mln7q"
Oct 30 00:31:50.843255 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.843051    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-host\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.843255 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.843057    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ef9a0a39-051f-45a1-acc9-354a23e5b00b-cni-binary-copy\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:50.843255 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.843071    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/05a614f5-3112-4216-b894-68bf3945458c-hosts-file\") pod \"node-resolver-fqtzs\" (UID: \"05a614f5-3112-4216-b894-68bf3945458c\") " pod="openshift-dns/node-resolver-fqtzs"
Oct 30 00:31:50.843255 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.843087    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b734765-8e7f-4d8e-a74b-bd179adf33ee-cni-binary-copy\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.843255 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.843102    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3b734765-8e7f-4d8e-a74b-bd179adf33ee-multus-daemon-config\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.843255 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.843113    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-env-overrides\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.843255 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.843214    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ef9a0a39-051f-45a1-acc9-354a23e5b00b-cnibin\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:50.843255 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.843249    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-sys\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.844107 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.843274    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-lib-modules\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.844107 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.843278    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b734765-8e7f-4d8e-a74b-bd179adf33ee-cni-binary-copy\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.844107 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.843296    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-cnibin\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.844107 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.843340    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-env-overrides\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.844107 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.843356    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-host-run-netns\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.844107 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.843401    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-cni-netd\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.844107 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.843439    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xvts\" (UniqueName: \"kubernetes.io/projected/b3705749-d093-4b49-878d-e4616e81a450-kube-api-access-7xvts\") pod \"network-check-target-wpvjg\" (UID: \"b3705749-d093-4b49-878d-e4616e81a450\") " pod="openshift-network-diagnostics/network-check-target-wpvjg"
Oct 30 00:31:50.844107 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.843465    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-crio-socket\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:50.844107 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.843491    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e552ba30-5b09-48a7-bffb-6a65a094c2bc-metrics-certs\") pod \"network-metrics-daemon-jldmt\" (UID: \"e552ba30-5b09-48a7-bffb-6a65a094c2bc\") " pod="openshift-multus/network-metrics-daemon-jldmt"
Oct 30 00:31:50.844107 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.843519    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-host-run-k8s-cni-cncf-io\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.844107 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.843555    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-host-var-lib-cni-multus\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.844107 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.843579    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-systemd-units\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.844107 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.843609    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4f7ca654-edeb-4457-9239-8da741b5dafd-agent-certs\") pod \"konnectivity-agent-6wzd9\" (UID: \"4f7ca654-edeb-4457-9239-8da741b5dafd\") " pod="kube-system/konnectivity-agent-6wzd9"
Oct 30 00:31:50.844107 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.843657    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ef9a0a39-051f-45a1-acc9-354a23e5b00b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:50.844107 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.843706    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-run\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.844107 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.843746    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-var-lib-kubelet\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.844107 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.843823    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4f7ca654-edeb-4457-9239-8da741b5dafd-agent-certs\") pod \"konnectivity-agent-6wzd9\" (UID: \"4f7ca654-edeb-4457-9239-8da741b5dafd\") " pod="kube-system/konnectivity-agent-6wzd9"
Oct 30 00:31:50.844107 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.843842    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:50.844107 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.843872    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-host-run-multus-certs\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.844107 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.843898    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-etc-kubernetes\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.844107 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.843921    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-etc-sysctl-d\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.844107 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.843963    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-ovnkube-config\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.844107 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.844006    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgw94\" (UniqueName: \"kubernetes.io/projected/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-kube-api-access-bgw94\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.844107 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.844018    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:50.844107 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.844066    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5687019d-945a-4307-8269-5c0ae6f377ec-tmp\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.844107 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.844084    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30e43c48-92a7-4c68-887c-9aaf7d6d1e1b-host\") pod \"node-ca-9hslh\" (UID: \"30e43c48-92a7-4c68-887c-9aaf7d6d1e1b\") " pod="openshift-image-registry/node-ca-9hslh"
Oct 30 00:31:50.845150 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.844124    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-etc-kubernetes\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.845150 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.844135    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-ovnkube-config\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.845150 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.844149    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5687019d-945a-4307-8269-5c0ae6f377ec-tmp\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.845150 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.844508    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.845150 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.844573    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-device-dir\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:50.845150 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.844610    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-config-volume\") pod \"dns-default-9ml2g\" (UID: \"b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f\") " pod="openshift-dns/dns-default-9ml2g"
Oct 30 00:31:50.845150 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.844746    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tntlm\" (UniqueName: \"kubernetes.io/projected/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-kube-api-access-tntlm\") pod \"dns-default-9ml2g\" (UID: \"b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f\") " pod="openshift-dns/dns-default-9ml2g"
Oct 30 00:31:50.845150 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.844807    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-host-var-lib-kubelet\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.845150 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.844886    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-config-volume\") pod \"dns-default-9ml2g\" (UID: \"b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f\") " pod="openshift-dns/dns-default-9ml2g"
Oct 30 00:31:50.845526 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.845160    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5mw6\" (UniqueName: \"kubernetes.io/projected/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-kube-api-access-g5mw6\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:50.845526 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.845241    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4a99683d-3f83-465d-9152-2f422641e087-iptables-alerter-script\") pod \"iptables-alerter-s72l8\" (UID: \"4a99683d-3f83-465d-9152-2f422641e087\") " pod="openshift-network-operator/iptables-alerter-s72l8"
Oct 30 00:31:50.845526 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.845281    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-etc-openvswitch\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.845526 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.845313    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-etc-modprobe-d\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.845526 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.845342    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-etc-sysctl-conf\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.845526 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.845374    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7rhg\" (UniqueName: \"kubernetes.io/projected/3b734765-8e7f-4d8e-a74b-bd179adf33ee-kube-api-access-k7rhg\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.845526 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.845400    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdxj7\" (UniqueName: \"kubernetes.io/projected/a20a733f-4947-4a14-9db0-a5be23cb9ccb-kube-api-access-fdxj7\") pod \"ingress-canary-jb5sp\" (UID: \"a20a733f-4947-4a14-9db0-a5be23cb9ccb\") " pod="openshift-ingress-canary/ingress-canary-jb5sp"
Oct 30 00:31:50.845526 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.845432    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-run-systemd\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.845526 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.845462    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-var-lib-openvswitch\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.845526 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.845491    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-multus-cni-dir\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.845526 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.845525    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b0df054-0749-4417-bb8b-b12b3353d25e-original-pull-secret\") pod \"global-pull-secret-syncer-f27vk\" (UID: \"3b0df054-0749-4417-bb8b-b12b3353d25e\") " pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:50.846004 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.845555    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-cni-bin\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.846004 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.845578    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-sys-fs\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:50.846004 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.845607    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-tmp-dir\") pod \"dns-default-9ml2g\" (UID: \"b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f\") " pod="openshift-dns/dns-default-9ml2g"
Oct 30 00:31:50.846004 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.845638    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ef9a0a39-051f-45a1-acc9-354a23e5b00b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:50.846004 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.845669    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpfcq\" (UniqueName: \"kubernetes.io/projected/e552ba30-5b09-48a7-bffb-6a65a094c2bc-kube-api-access-gpfcq\") pod \"network-metrics-daemon-jldmt\" (UID: \"e552ba30-5b09-48a7-bffb-6a65a094c2bc\") " pod="openshift-multus/network-metrics-daemon-jldmt"
Oct 30 00:31:50.846004 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.845693    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a99683d-3f83-465d-9152-2f422641e087-host-slash\") pod \"iptables-alerter-s72l8\" (UID: \"4a99683d-3f83-465d-9152-2f422641e087\") " pod="openshift-network-operator/iptables-alerter-s72l8"
Oct 30 00:31:50.846004 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.845726    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:50.846004 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.845754    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-socket-dir\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:50.846338 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.846245    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4a99683d-3f83-465d-9152-2f422641e087-iptables-alerter-script\") pod \"iptables-alerter-s72l8\" (UID: \"4a99683d-3f83-465d-9152-2f422641e087\") " pod="openshift-network-operator/iptables-alerter-s72l8"
Oct 30 00:31:50.846693 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.846671    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-tmp-dir\") pod \"dns-default-9ml2g\" (UID: \"b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f\") " pod="openshift-dns/dns-default-9ml2g"
Oct 30 00:31:50.846982 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.846962    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b0df054-0749-4417-bb8b-b12b3353d25e-original-pull-secret\") pod \"global-pull-secret-syncer-f27vk\" (UID: \"3b0df054-0749-4417-bb8b-b12b3353d25e\") " pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:50.847092 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.847072    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ef9a0a39-051f-45a1-acc9-354a23e5b00b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:50.863641 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.863603    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzcz2\" (UniqueName: \"kubernetes.io/projected/30e43c48-92a7-4c68-887c-9aaf7d6d1e1b-kube-api-access-zzcz2\") pod \"node-ca-9hslh\" (UID: \"30e43c48-92a7-4c68-887c-9aaf7d6d1e1b\") " pod="openshift-image-registry/node-ca-9hslh"
Oct 30 00:31:50.863898 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.863876    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28lgt\" (UniqueName: \"kubernetes.io/projected/4a99683d-3f83-465d-9152-2f422641e087-kube-api-access-28lgt\") pod \"iptables-alerter-s72l8\" (UID: \"4a99683d-3f83-465d-9152-2f422641e087\") " pod="openshift-network-operator/iptables-alerter-s72l8"
Oct 30 00:31:50.864037 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.864018    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqzmf\" (UniqueName: \"kubernetes.io/projected/5687019d-945a-4307-8269-5c0ae6f377ec-kube-api-access-gqzmf\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.864166 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.864150    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pchtx\" (UniqueName: \"kubernetes.io/projected/05a614f5-3112-4216-b894-68bf3945458c-kube-api-access-pchtx\") pod \"node-resolver-fqtzs\" (UID: \"05a614f5-3112-4216-b894-68bf3945458c\") " pod="openshift-dns/node-resolver-fqtzs"
Oct 30 00:31:50.865979 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.865957    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wblgx\" (UniqueName: \"kubernetes.io/projected/ef9a0a39-051f-45a1-acc9-354a23e5b00b-kube-api-access-wblgx\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:50.866448 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.866427    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpfcq\" (UniqueName: \"kubernetes.io/projected/e552ba30-5b09-48a7-bffb-6a65a094c2bc-kube-api-access-gpfcq\") pod \"network-metrics-daemon-jldmt\" (UID: \"e552ba30-5b09-48a7-bffb-6a65a094c2bc\") " pod="openshift-multus/network-metrics-daemon-jldmt"
Oct 30 00:31:50.866448 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.866432    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bgt8\" (UniqueName: \"kubernetes.io/projected/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-kube-api-access-4bgt8\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:50.866699 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.866684    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xvts\" (UniqueName: \"kubernetes.io/projected/b3705749-d093-4b49-878d-e4616e81a450-kube-api-access-7xvts\") pod \"network-check-target-wpvjg\" (UID: \"b3705749-d093-4b49-878d-e4616e81a450\") " pod="openshift-network-diagnostics/network-check-target-wpvjg"
Oct 30 00:31:50.868550 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.868530    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tntlm\" (UniqueName: \"kubernetes.io/projected/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-kube-api-access-tntlm\") pod \"dns-default-9ml2g\" (UID: \"b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f\") " pod="openshift-dns/dns-default-9ml2g"
Oct 30 00:31:50.868909 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.868888    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7rhg\" (UniqueName: \"kubernetes.io/projected/3b734765-8e7f-4d8e-a74b-bd179adf33ee-kube-api-access-k7rhg\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.869005 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.868972    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgw94\" (UniqueName: \"kubernetes.io/projected/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-kube-api-access-bgw94\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.869005 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.868995    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4nwk\" (UniqueName: \"kubernetes.io/projected/8ff169ff-dc1a-4344-b19b-adaabe98f20d-kube-api-access-c4nwk\") pod \"service-ca-5fffc97bdf-mln7q\" (UID: \"8ff169ff-dc1a-4344-b19b-adaabe98f20d\") " pod="openshift-service-ca/service-ca-5fffc97bdf-mln7q"
Oct 30 00:31:50.869131 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.869109    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5mw6\" (UniqueName: \"kubernetes.io/projected/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-kube-api-access-g5mw6\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:50.870360 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.870342    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdxj7\" (UniqueName: \"kubernetes.io/projected/a20a733f-4947-4a14-9db0-a5be23cb9ccb-kube-api-access-fdxj7\") pod \"ingress-canary-jb5sp\" (UID: \"a20a733f-4947-4a14-9db0-a5be23cb9ccb\") " pod="openshift-ingress-canary/ingress-canary-jb5sp"
Oct 30 00:31:50.947030 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.946938    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-crio-socket\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:50.947030 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.946974    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e552ba30-5b09-48a7-bffb-6a65a094c2bc-metrics-certs\") pod \"network-metrics-daemon-jldmt\" (UID: \"e552ba30-5b09-48a7-bffb-6a65a094c2bc\") " pod="openshift-multus/network-metrics-daemon-jldmt"
Oct 30 00:31:50.947030 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.946996    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-host-run-k8s-cni-cncf-io\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.947030 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947017    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-host-var-lib-cni-multus\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.947345 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947041    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-systemd-units\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.947345 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947075    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-crio-socket\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:50.947345 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947089    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-host-var-lib-cni-multus\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.947345 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947110    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ef9a0a39-051f-45a1-acc9-354a23e5b00b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:50.947345 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947114    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-systemd-units\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.947345 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947084    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-host-run-k8s-cni-cncf-io\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.947345 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947152    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-run\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.947345 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947165    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ef9a0a39-051f-45a1-acc9-354a23e5b00b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:50.947345 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947188    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-var-lib-kubelet\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.947345 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947195    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-run\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.947345 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947214    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-host-run-multus-certs\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.947345 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947237    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-etc-kubernetes\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.947345 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947272    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-var-lib-kubelet\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.947345 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947275    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-host-run-multus-certs\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.947345 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947278    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-etc-kubernetes\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.947345 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947292    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-etc-sysctl-d\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.947345 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947319    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30e43c48-92a7-4c68-887c-9aaf7d6d1e1b-host\") pod \"node-ca-9hslh\" (UID: \"30e43c48-92a7-4c68-887c-9aaf7d6d1e1b\") " pod="openshift-image-registry/node-ca-9hslh"
Oct 30 00:31:50.947345 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947334    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-etc-kubernetes\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.947345 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947349    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947368    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30e43c48-92a7-4c68-887c-9aaf7d6d1e1b-host\") pod \"node-ca-9hslh\" (UID: \"30e43c48-92a7-4c68-887c-9aaf7d6d1e1b\") " pod="openshift-image-registry/node-ca-9hslh"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947370    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-device-dir\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947394    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-etc-kubernetes\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947404    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-device-dir\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947408    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-host-var-lib-kubelet\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947412    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-etc-sysctl-d\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947439    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-etc-openvswitch\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947444    4393 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947452    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947464    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-etc-modprobe-d\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947455    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-host-var-lib-kubelet\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947484    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-etc-openvswitch\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947489    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-etc-sysctl-conf\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947518    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-run-systemd\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947537    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-var-lib-openvswitch\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947543    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-etc-modprobe-d\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947589    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-run-systemd\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947600    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-etc-sysctl-conf\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947613    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-multus-cni-dir\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947626    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-var-lib-openvswitch\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947649    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-cni-bin\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947669    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-sys-fs\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947681    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-multus-cni-dir\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947686    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a99683d-3f83-465d-9152-2f422641e087-host-slash\") pod \"iptables-alerter-s72l8\" (UID: \"4a99683d-3f83-465d-9152-2f422641e087\") " pod="openshift-network-operator/iptables-alerter-s72l8"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947703    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947708    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-cni-bin\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947718    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-socket-dir\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947714    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-sys-fs\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947733    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a99683d-3f83-465d-9152-2f422641e087-host-slash\") pod \"iptables-alerter-s72l8\" (UID: \"4a99683d-3f83-465d-9152-2f422641e087\") " pod="openshift-network-operator/iptables-alerter-s72l8"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947744    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-etc-sysconfig\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947764    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3b0df054-0749-4417-bb8b-b12b3353d25e-dbus\") pod \"global-pull-secret-syncer-f27vk\" (UID: \"3b0df054-0749-4417-bb8b-b12b3353d25e\") " pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947766    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947801    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-run-netns\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947804    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-socket-dir\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947826    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-registration-dir\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947850    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3b0df054-0749-4417-bb8b-b12b3353d25e-dbus\") pod \"global-pull-secret-syncer-f27vk\" (UID: \"3b0df054-0749-4417-bb8b-b12b3353d25e\") " pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947863    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-run-netns\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947851    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-etc-selinux\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947871    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-registration-dir\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947828    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-etc-sysconfig\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947898    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc-etc-selinux\") pod \"aws-ebs-csi-driver-node-rfkwz\" (UID: \"e3b48e29-13f0-4f4d-a19c-ddaa7e2431dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rfkwz"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947905    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-metrics-tls\") pod \"dns-default-9ml2g\" (UID: \"b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f\") " pod="openshift-dns/dns-default-9ml2g"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947932    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-run-openvswitch\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947965    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-run-openvswitch\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.947959    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-run-ovn\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.948091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948058    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-run-ovn\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948148    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ef9a0a39-051f-45a1-acc9-354a23e5b00b-system-cni-dir\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948176    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948201    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-system-cni-dir\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948218    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ef9a0a39-051f-45a1-acc9-354a23e5b00b-system-cni-dir\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948251    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-node-log\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948284    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-system-cni-dir\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948292    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-etc-systemd\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:31:50.948283    4393 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948329    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-host-var-lib-cni-bin\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948342    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-etc-systemd\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948356    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3b0df054-0749-4417-bb8b-b12b3353d25e-kubelet-config\") pod \"global-pull-secret-syncer-f27vk\" (UID: \"3b0df054-0749-4417-bb8b-b12b3353d25e\") " pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948358    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-node-log\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948383    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-host-var-lib-cni-bin\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:31:50.948399    4393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-insights-runtime-extractor-tls podName:626ec6c0-d616-40ae-bf8b-4e8a927c31ac nodeName:}" failed. No retries permitted until 2026-10-30 00:31:51.448373819 +0000 UTC m=+1.836050205 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jbmfs" (UID: "626ec6c0-d616-40ae-bf8b-4e8a927c31ac") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948418    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-slash\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948447    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-log-socket\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948452    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-slash\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948406    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3b0df054-0749-4417-bb8b-b12b3353d25e-kubelet-config\") pod \"global-pull-secret-syncer-f27vk\" (UID: \"3b0df054-0749-4417-bb8b-b12b3353d25e\") " pod="kube-system/global-pull-secret-syncer-f27vk"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948481    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-run-ovn-kubernetes\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948492    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-log-socket\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948516    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-kubelet\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948544    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-multus-socket-dir-parent\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948572    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-hostroot\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948573    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-run-ovn-kubernetes\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948592    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-multus-conf-dir\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948614    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-kubelet\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948616    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-host\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948634    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-multus-socket-dir-parent\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948650    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/05a614f5-3112-4216-b894-68bf3945458c-hosts-file\") pod \"node-resolver-fqtzs\" (UID: \"05a614f5-3112-4216-b894-68bf3945458c\") " pod="openshift-dns/node-resolver-fqtzs"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948675    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ef9a0a39-051f-45a1-acc9-354a23e5b00b-cnibin\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948673    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-host\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948690    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-multus-conf-dir\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948711    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ef9a0a39-051f-45a1-acc9-354a23e5b00b-cnibin\") pod \"multus-additional-cni-plugins-csc84\" (UID: \"ef9a0a39-051f-45a1-acc9-354a23e5b00b\") " pod="openshift-multus/multus-additional-cni-plugins-csc84"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948725    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-hostroot\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948722    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/05a614f5-3112-4216-b894-68bf3945458c-hosts-file\") pod \"node-resolver-fqtzs\" (UID: \"05a614f5-3112-4216-b894-68bf3945458c\") " pod="openshift-dns/node-resolver-fqtzs"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948766    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-sys\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948817    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-lib-modules\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948842    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-cnibin\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948850    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-sys\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948867    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-host-run-netns\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948921    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-host-run-netns\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948924    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b734765-8e7f-4d8e-a74b-bd179adf33ee-cnibin\") pod \"multus-r58sp\" (UID: \"3b734765-8e7f-4d8e-a74b-bd179adf33ee\") " pod="openshift-multus/multus-r58sp"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948960    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5687019d-945a-4307-8269-5c0ae6f377ec-lib-modules\") pod \"tuned-lnnlc\" (UID: \"5687019d-945a-4307-8269-5c0ae6f377ec\") " pod="openshift-cluster-node-tuning-operator/tuned-lnnlc"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948959    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-cni-netd\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.949624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.948989    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835-host-cni-netd\") pod \"ovnkube-node-pxfhp\" (UID: \"2bdb8d7d-1c89-49ae-ab3b-5b7bbe2c1835\") " pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:50.951704 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.950029    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e552ba30-5b09-48a7-bffb-6a65a094c2bc-metrics-certs\") pod \"network-metrics-daemon-jldmt\" (UID: \"e552ba30-5b09-48a7-bffb-6a65a094c2bc\") " pod="openshift-multus/network-metrics-daemon-jldmt"
Oct 30 00:31:50.951704 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:50.950252    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f-metrics-tls\") pod \"dns-default-9ml2g\" (UID: \"b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f\") " pod="openshift-dns/dns-default-9ml2g"
Oct 30 00:31:51.021044 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:51.021015    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jldmt"
Oct 30 00:31:51.022038 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:51.022019    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9ml2g"
Oct 30 00:31:51.347133 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:51.347048    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jldmt"]
Oct 30 00:31:51.350451 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:51.350415    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode552ba30_5b09_48a7_bffb_6a65a094c2bc.slice/crio-03f1f8e44a7c002d2ff857b9ee6a54f90a60c7810ac45775b0518ffbe53a407a WatchSource:0}: Error finding container 03f1f8e44a7c002d2ff857b9ee6a54f90a60c7810ac45775b0518ffbe53a407a: Status 404 returned error can't find the container with id 03f1f8e44a7c002d2ff857b9ee6a54f90a60c7810ac45775b0518ffbe53a407a
Oct 30 00:31:51.352728 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:51.352710    4393 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:31:51.378395 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:51.378370    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9ml2g"]
Oct 30 00:31:51.380830 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:51.380804    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3aa0462_2e3d_4df7_9632_2f1c28ad6d6f.slice/crio-c4f71116b9784f6f28e241aebd164591e1496906bd9629eefdb4340ca466e23a WatchSource:0}: Error finding container c4f71116b9784f6f28e241aebd164591e1496906bd9629eefdb4340ca466e23a: Status 404 returned error can't find the container with id c4f71116b9784f6f28e241aebd164591e1496906bd9629eefdb4340ca466e23a
Oct 30 00:31:51.451715 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:51.451670    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:51.451895 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:31:51.451840    4393 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:31:51.451933 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:31:51.451906    4393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-insights-runtime-extractor-tls podName:626ec6c0-d616-40ae-bf8b-4e8a927c31ac nodeName:}" failed. No retries permitted until 2026-10-30 00:31:52.451889477 +0000 UTC m=+2.839565844 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jbmfs" (UID: "626ec6c0-d616-40ae-bf8b-4e8a927c31ac") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:31:51.761921 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:51.761880    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jldmt" event={"ID":"e552ba30-5b09-48a7-bffb-6a65a094c2bc","Type":"ContainerStarted","Data":"03f1f8e44a7c002d2ff857b9ee6a54f90a60c7810ac45775b0518ffbe53a407a"}
Oct 30 00:31:51.763059 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:51.763028    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9ml2g" event={"ID":"b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f","Type":"ContainerStarted","Data":"c4f71116b9784f6f28e241aebd164591e1496906bd9629eefdb4340ca466e23a"}
Oct 30 00:31:51.793203 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:51.792802    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wpvjg"
Oct 30 00:31:52.459617 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:52.459563    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:52.459843 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:31:52.459749    4393 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 00:31:52.459916 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:31:52.459868    4393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-insights-runtime-extractor-tls podName:626ec6c0-d616-40ae-bf8b-4e8a927c31ac nodeName:}" failed. No retries permitted until 2026-10-30 00:31:54.459846624 +0000 UTC m=+4.847523004 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jbmfs" (UID: "626ec6c0-d616-40ae-bf8b-4e8a927c31ac") : secret "insights-runtime-extractor-tls" not found
Oct 30 00:31:53.773803 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:53.773748    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jldmt" event={"ID":"e552ba30-5b09-48a7-bffb-6a65a094c2bc","Type":"ContainerStarted","Data":"548f35120ae519b416869540c169c9d2a9871ceb0ea8d055e04b8d727d5813bf"}
Oct 30 00:31:53.774289 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:53.773834    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jldmt" event={"ID":"e552ba30-5b09-48a7-bffb-6a65a094c2bc","Type":"ContainerStarted","Data":"63b90b27bf17f64741d254f88597fee422be7576aee038b46e2b1c0112a85d4a"}
Oct 30 00:31:53.776000 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:53.775969    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9ml2g" event={"ID":"b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f","Type":"ContainerStarted","Data":"43c14dee964376b0f39e40e3fa61a88f92089a6ab6b23ae2cd824b8b874168f2"}
Oct 30 00:31:54.472870 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:54.472823    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:54.475559 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:54.475526    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/626ec6c0-d616-40ae-bf8b-4e8a927c31ac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jbmfs\" (UID: \"626ec6c0-d616-40ae-bf8b-4e8a927c31ac\") " pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:54.622007 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:54.621975    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jbmfs"
Oct 30 00:31:54.781936 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:54.781846    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9ml2g" event={"ID":"b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f","Type":"ContainerStarted","Data":"a0c1d857281b52ed9afe80c73791b907ea72bd08f7d3c4089a176de83aca5a83"}
Oct 30 00:31:54.782333 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:54.782103    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9ml2g"
Oct 30 00:31:54.964599 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:54.964561    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jbmfs"]
Oct 30 00:31:54.965612 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:31:54.965580    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod626ec6c0_d616_40ae_bf8b_4e8a927c31ac.slice/crio-a6cc57315581f8b22ac2babf863b8788ba884d9026725a32cc00cd02af563ba5 WatchSource:0}: Error finding container a6cc57315581f8b22ac2babf863b8788ba884d9026725a32cc00cd02af563ba5: Status 404 returned error can't find the container with id a6cc57315581f8b22ac2babf863b8788ba884d9026725a32cc00cd02af563ba5
Oct 30 00:31:55.785940 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:55.785907    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jbmfs" event={"ID":"626ec6c0-d616-40ae-bf8b-4e8a927c31ac","Type":"ContainerStarted","Data":"e3884a59894fcc901c0e2d508379f80b1520859c9b224b7bb4794f6ab1004361"}
Oct 30 00:31:55.786345 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:55.785964    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jbmfs" event={"ID":"626ec6c0-d616-40ae-bf8b-4e8a927c31ac","Type":"ContainerStarted","Data":"a6cc57315581f8b22ac2babf863b8788ba884d9026725a32cc00cd02af563ba5"}
Oct 30 00:31:56.791397 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:56.791361    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jbmfs" event={"ID":"626ec6c0-d616-40ae-bf8b-4e8a927c31ac","Type":"ContainerStarted","Data":"a134071c0399fd433b7f80691587a47135236373c4c9bae4d24e42074e8c09d4"}
Oct 30 00:31:57.797048 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:57.796961    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jbmfs" event={"ID":"626ec6c0-d616-40ae-bf8b-4e8a927c31ac","Type":"ContainerStarted","Data":"84a43bb2fb3389b6b5cb5a73bb221803f6032d5bba108d3066e6c28cd86372f8"}
Oct 30 00:31:59.289212 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:59.289186    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:59.858175 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:59.858141    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:31:59.858360 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:31:59.858343    4393 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 00:32:00.032280 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.032242    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-p5zpd"]
Oct 30 00:32:00.034112 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.034075    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-r7bf7"]
Oct 30 00:32:00.034246 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.034199    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-p5zpd"
Oct 30 00:32:00.035760 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.035741    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7cc457646f-sjj89"]
Oct 30 00:32:00.035913 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.035892    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-r7bf7"
Oct 30 00:32:00.037501 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.037485    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cc457646f-sjj89"
Oct 30 00:32:00.073568 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.073537    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 00:32:00.078043 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.078001    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 00:32:00.078223 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.078147    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 00:32:00.078223 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.078205    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 00:32:00.078324 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.078239    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 00:32:00.082678 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.078155    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-m5q4g\""
Oct 30 00:32:00.082678 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.078895    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 00:32:00.082678 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.078724    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-w4ngz\""
Oct 30 00:32:00.082678 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.079033    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 00:32:00.082678 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.079308    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-6p4dh\""
Oct 30 00:32:00.082678 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.079463    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 00:32:00.082678 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.081715    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-r7bf7"]
Oct 30 00:32:00.082678 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.081879    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cc457646f-sjj89"]
Oct 30 00:32:00.082678 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.081912    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-p5zpd"]
Oct 30 00:32:00.108127 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.108096    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72a65fa9-4477-434a-aaac-0394505927e5-service-ca\") pod \"console-7cc457646f-sjj89\" (UID: \"72a65fa9-4477-434a-aaac-0394505927e5\") " pod="openshift-console/console-7cc457646f-sjj89"
Oct 30 00:32:00.108338 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.108132    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72a65fa9-4477-434a-aaac-0394505927e5-console-serving-cert\") pod \"console-7cc457646f-sjj89\" (UID: \"72a65fa9-4477-434a-aaac-0394505927e5\") " pod="openshift-console/console-7cc457646f-sjj89"
Oct 30 00:32:00.108338 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.108163    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a3d031e-b63e-4201-afb9-ec659b67ac1e-tmp\") pod \"downloads-6dd789c8b7-r7bf7\" (UID: \"4a3d031e-b63e-4201-afb9-ec659b67ac1e\") " pod="openshift-console/downloads-6dd789c8b7-r7bf7"
Oct 30 00:32:00.108338 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.108181    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm5vw\" (UniqueName: \"kubernetes.io/projected/4a3d031e-b63e-4201-afb9-ec659b67ac1e-kube-api-access-lm5vw\") pod \"downloads-6dd789c8b7-r7bf7\" (UID: \"4a3d031e-b63e-4201-afb9-ec659b67ac1e\") " pod="openshift-console/downloads-6dd789c8b7-r7bf7"
Oct 30 00:32:00.108338 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.108244    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72a65fa9-4477-434a-aaac-0394505927e5-console-config\") pod \"console-7cc457646f-sjj89\" (UID: \"72a65fa9-4477-434a-aaac-0394505927e5\") " pod="openshift-console/console-7cc457646f-sjj89"
Oct 30 00:32:00.108488 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.108350    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72a65fa9-4477-434a-aaac-0394505927e5-oauth-serving-cert\") pod \"console-7cc457646f-sjj89\" (UID: \"72a65fa9-4477-434a-aaac-0394505927e5\") " pod="openshift-console/console-7cc457646f-sjj89"
Oct 30 00:32:00.108488 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.108377    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9cbk\" (UniqueName: \"kubernetes.io/projected/72a65fa9-4477-434a-aaac-0394505927e5-kube-api-access-l9cbk\") pod \"console-7cc457646f-sjj89\" (UID: \"72a65fa9-4477-434a-aaac-0394505927e5\") " pod="openshift-console/console-7cc457646f-sjj89"
Oct 30 00:32:00.108488 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.108395    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7258c681-0692-4203-b3da-a3bab6b298dd-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-p5zpd\" (UID: \"7258c681-0692-4203-b3da-a3bab6b298dd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-p5zpd"
Oct 30 00:32:00.108488 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.108425    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72a65fa9-4477-434a-aaac-0394505927e5-console-oauth-config\") pod \"console-7cc457646f-sjj89\" (UID: \"72a65fa9-4477-434a-aaac-0394505927e5\") " pod="openshift-console/console-7cc457646f-sjj89"
Oct 30 00:32:00.108488 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.108441    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72a65fa9-4477-434a-aaac-0394505927e5-tmp\") pod \"console-7cc457646f-sjj89\" (UID: \"72a65fa9-4477-434a-aaac-0394505927e5\") " pod="openshift-console/console-7cc457646f-sjj89"
Oct 30 00:32:00.148558 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.148528    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7d899fbfd7-2j85r"]
Oct 30 00:32:00.150382 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.150366    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d899fbfd7-2j85r"
Oct 30 00:32:00.174571 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.174542    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 00:32:00.174571 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.174542    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 00:32:00.174571 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.174578    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 00:32:00.174862 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.174629    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 00:32:00.174862 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.174674    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-j8zc4\""
Oct 30 00:32:00.180290 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.180267    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d899fbfd7-2j85r"]
Oct 30 00:32:00.209383 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.209357    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72a65fa9-4477-434a-aaac-0394505927e5-service-ca\") pod \"console-7cc457646f-sjj89\" (UID: \"72a65fa9-4477-434a-aaac-0394505927e5\") " pod="openshift-console/console-7cc457646f-sjj89"
Oct 30 00:32:00.209568 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.209395    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72a65fa9-4477-434a-aaac-0394505927e5-console-serving-cert\") pod \"console-7cc457646f-sjj89\" (UID: \"72a65fa9-4477-434a-aaac-0394505927e5\") " pod="openshift-console/console-7cc457646f-sjj89"
Oct 30 00:32:00.209568 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.209431    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a3d031e-b63e-4201-afb9-ec659b67ac1e-tmp\") pod \"downloads-6dd789c8b7-r7bf7\" (UID: \"4a3d031e-b63e-4201-afb9-ec659b67ac1e\") " pod="openshift-console/downloads-6dd789c8b7-r7bf7"
Oct 30 00:32:00.209568 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.209448    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm5vw\" (UniqueName: \"kubernetes.io/projected/4a3d031e-b63e-4201-afb9-ec659b67ac1e-kube-api-access-lm5vw\") pod \"downloads-6dd789c8b7-r7bf7\" (UID: \"4a3d031e-b63e-4201-afb9-ec659b67ac1e\") " pod="openshift-console/downloads-6dd789c8b7-r7bf7"
Oct 30 00:32:00.209568 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.209468    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72a65fa9-4477-434a-aaac-0394505927e5-console-config\") pod \"console-7cc457646f-sjj89\" (UID: \"72a65fa9-4477-434a-aaac-0394505927e5\") " pod="openshift-console/console-7cc457646f-sjj89"
Oct 30 00:32:00.209568 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.209493    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72a65fa9-4477-434a-aaac-0394505927e5-oauth-serving-cert\") pod \"console-7cc457646f-sjj89\" (UID: \"72a65fa9-4477-434a-aaac-0394505927e5\") " pod="openshift-console/console-7cc457646f-sjj89"
Oct 30 00:32:00.209568 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.209529    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9cbk\" (UniqueName: \"kubernetes.io/projected/72a65fa9-4477-434a-aaac-0394505927e5-kube-api-access-l9cbk\") pod \"console-7cc457646f-sjj89\" (UID: \"72a65fa9-4477-434a-aaac-0394505927e5\") " pod="openshift-console/console-7cc457646f-sjj89"
Oct 30 00:32:00.209568 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.209557    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7258c681-0692-4203-b3da-a3bab6b298dd-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-p5zpd\" (UID: \"7258c681-0692-4203-b3da-a3bab6b298dd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-p5zpd"
Oct 30 00:32:00.210182 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.209588    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72a65fa9-4477-434a-aaac-0394505927e5-console-oauth-config\") pod \"console-7cc457646f-sjj89\" (UID: \"72a65fa9-4477-434a-aaac-0394505927e5\") " pod="openshift-console/console-7cc457646f-sjj89"
Oct 30 00:32:00.210182 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.209834    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72a65fa9-4477-434a-aaac-0394505927e5-tmp\") pod \"console-7cc457646f-sjj89\" (UID: \"72a65fa9-4477-434a-aaac-0394505927e5\") " pod="openshift-console/console-7cc457646f-sjj89"
Oct 30 00:32:00.210182 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.210121    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a3d031e-b63e-4201-afb9-ec659b67ac1e-tmp\") pod \"downloads-6dd789c8b7-r7bf7\" (UID: \"4a3d031e-b63e-4201-afb9-ec659b67ac1e\") " pod="openshift-console/downloads-6dd789c8b7-r7bf7"
Oct 30 00:32:00.210366 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.210342    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72a65fa9-4477-434a-aaac-0394505927e5-console-config\") pod \"console-7cc457646f-sjj89\" (UID: \"72a65fa9-4477-434a-aaac-0394505927e5\") " pod="openshift-console/console-7cc457646f-sjj89"
Oct 30 00:32:00.210456 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.210342    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72a65fa9-4477-434a-aaac-0394505927e5-service-ca\") pod \"console-7cc457646f-sjj89\" (UID: \"72a65fa9-4477-434a-aaac-0394505927e5\") " pod="openshift-console/console-7cc457646f-sjj89"
Oct 30 00:32:00.210456 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.210426    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72a65fa9-4477-434a-aaac-0394505927e5-tmp\") pod \"console-7cc457646f-sjj89\" (UID: \"72a65fa9-4477-434a-aaac-0394505927e5\") " pod="openshift-console/console-7cc457646f-sjj89"
Oct 30 00:32:00.210802 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.210758    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72a65fa9-4477-434a-aaac-0394505927e5-oauth-serving-cert\") pod \"console-7cc457646f-sjj89\" (UID: \"72a65fa9-4477-434a-aaac-0394505927e5\") " pod="openshift-console/console-7cc457646f-sjj89"
Oct 30 00:32:00.212239 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.212209    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72a65fa9-4477-434a-aaac-0394505927e5-console-serving-cert\") pod \"console-7cc457646f-sjj89\" (UID: \"72a65fa9-4477-434a-aaac-0394505927e5\") " pod="openshift-console/console-7cc457646f-sjj89"
Oct 30 00:32:00.212649 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.212627    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72a65fa9-4477-434a-aaac-0394505927e5-console-oauth-config\") pod \"console-7cc457646f-sjj89\" (UID: \"72a65fa9-4477-434a-aaac-0394505927e5\") " pod="openshift-console/console-7cc457646f-sjj89"
Oct 30 00:32:00.212699 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.212671    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7258c681-0692-4203-b3da-a3bab6b298dd-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-p5zpd\" (UID: \"7258c681-0692-4203-b3da-a3bab6b298dd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-p5zpd"
Oct 30 00:32:00.232102 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.232078    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9cbk\" (UniqueName: \"kubernetes.io/projected/72a65fa9-4477-434a-aaac-0394505927e5-kube-api-access-l9cbk\") pod \"console-7cc457646f-sjj89\" (UID: \"72a65fa9-4477-434a-aaac-0394505927e5\") " pod="openshift-console/console-7cc457646f-sjj89"
Oct 30 00:32:00.232224 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.232082    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm5vw\" (UniqueName: \"kubernetes.io/projected/4a3d031e-b63e-4201-afb9-ec659b67ac1e-kube-api-access-lm5vw\") pod \"downloads-6dd789c8b7-r7bf7\" (UID: \"4a3d031e-b63e-4201-afb9-ec659b67ac1e\") " pod="openshift-console/downloads-6dd789c8b7-r7bf7"
Oct 30 00:32:00.310493 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.310455    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21e682d8-71fc-4a64-9eda-1b5ed1788fde-ca-trust-extracted\") pod \"image-registry-7d899fbfd7-2j85r\" (UID: \"21e682d8-71fc-4a64-9eda-1b5ed1788fde\") " pod="openshift-image-registry/image-registry-7d899fbfd7-2j85r"
Oct 30 00:32:00.310493 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.310498    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21e682d8-71fc-4a64-9eda-1b5ed1788fde-image-registry-private-configuration\") pod \"image-registry-7d899fbfd7-2j85r\" (UID: \"21e682d8-71fc-4a64-9eda-1b5ed1788fde\") " pod="openshift-image-registry/image-registry-7d899fbfd7-2j85r"
Oct 30 00:32:00.311036 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.310524    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n2hf\" (UniqueName: \"kubernetes.io/projected/21e682d8-71fc-4a64-9eda-1b5ed1788fde-kube-api-access-8n2hf\") pod \"image-registry-7d899fbfd7-2j85r\" (UID: \"21e682d8-71fc-4a64-9eda-1b5ed1788fde\") " pod="openshift-image-registry/image-registry-7d899fbfd7-2j85r"
Oct 30 00:32:00.311036 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.310630    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21e682d8-71fc-4a64-9eda-1b5ed1788fde-registry-tls\") pod \"image-registry-7d899fbfd7-2j85r\" (UID: \"21e682d8-71fc-4a64-9eda-1b5ed1788fde\") " pod="openshift-image-registry/image-registry-7d899fbfd7-2j85r"
Oct 30 00:32:00.311036 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.310665    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21e682d8-71fc-4a64-9eda-1b5ed1788fde-bound-sa-token\") pod \"image-registry-7d899fbfd7-2j85r\" (UID: \"21e682d8-71fc-4a64-9eda-1b5ed1788fde\") " pod="openshift-image-registry/image-registry-7d899fbfd7-2j85r"
Oct 30 00:32:00.311036 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.310692    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21e682d8-71fc-4a64-9eda-1b5ed1788fde-installation-pull-secrets\") pod \"image-registry-7d899fbfd7-2j85r\" (UID: \"21e682d8-71fc-4a64-9eda-1b5ed1788fde\") " pod="openshift-image-registry/image-registry-7d899fbfd7-2j85r"
Oct 30 00:32:00.311036 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.310809    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21e682d8-71fc-4a64-9eda-1b5ed1788fde-trusted-ca\") pod \"image-registry-7d899fbfd7-2j85r\" (UID: \"21e682d8-71fc-4a64-9eda-1b5ed1788fde\") " pod="openshift-image-registry/image-registry-7d899fbfd7-2j85r"
Oct 30 00:32:00.311036 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.310847    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21e682d8-71fc-4a64-9eda-1b5ed1788fde-registry-certificates\") pod \"image-registry-7d899fbfd7-2j85r\" (UID: \"21e682d8-71fc-4a64-9eda-1b5ed1788fde\") " pod="openshift-image-registry/image-registry-7d899fbfd7-2j85r"
Oct 30 00:32:00.346188 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.346157    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-p5zpd"
Oct 30 00:32:00.357055 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.356716    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-r7bf7"
Oct 30 00:32:00.357830 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.357806    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cc457646f-sjj89"
Oct 30 00:32:00.411765 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.411617    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21e682d8-71fc-4a64-9eda-1b5ed1788fde-ca-trust-extracted\") pod \"image-registry-7d899fbfd7-2j85r\" (UID: \"21e682d8-71fc-4a64-9eda-1b5ed1788fde\") " pod="openshift-image-registry/image-registry-7d899fbfd7-2j85r"
Oct 30 00:32:00.411765 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.411665    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21e682d8-71fc-4a64-9eda-1b5ed1788fde-image-registry-private-configuration\") pod \"image-registry-7d899fbfd7-2j85r\" (UID: \"21e682d8-71fc-4a64-9eda-1b5ed1788fde\") " pod="openshift-image-registry/image-registry-7d899fbfd7-2j85r"
Oct 30 00:32:00.411765 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.411698    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n2hf\" (UniqueName: \"kubernetes.io/projected/21e682d8-71fc-4a64-9eda-1b5ed1788fde-kube-api-access-8n2hf\") pod \"image-registry-7d899fbfd7-2j85r\" (UID: \"21e682d8-71fc-4a64-9eda-1b5ed1788fde\") " pod="openshift-image-registry/image-registry-7d899fbfd7-2j85r"
Oct 30 00:32:00.411765 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.411725    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21e682d8-71fc-4a64-9eda-1b5ed1788fde-registry-tls\") pod \"image-registry-7d899fbfd7-2j85r\" (UID: \"21e682d8-71fc-4a64-9eda-1b5ed1788fde\") " pod="openshift-image-registry/image-registry-7d899fbfd7-2j85r"
Oct 30 00:32:00.411765 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.411755    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21e682d8-71fc-4a64-9eda-1b5ed1788fde-bound-sa-token\") pod \"image-registry-7d899fbfd7-2j85r\" (UID: \"21e682d8-71fc-4a64-9eda-1b5ed1788fde\") " pod="openshift-image-registry/image-registry-7d899fbfd7-2j85r"
Oct 30 00:32:00.412102 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.411811    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21e682d8-71fc-4a64-9eda-1b5ed1788fde-installation-pull-secrets\") pod \"image-registry-7d899fbfd7-2j85r\" (UID: \"21e682d8-71fc-4a64-9eda-1b5ed1788fde\") " pod="openshift-image-registry/image-registry-7d899fbfd7-2j85r"
Oct 30 00:32:00.412102 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.411840    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21e682d8-71fc-4a64-9eda-1b5ed1788fde-trusted-ca\") pod \"image-registry-7d899fbfd7-2j85r\" (UID: \"21e682d8-71fc-4a64-9eda-1b5ed1788fde\") " pod="openshift-image-registry/image-registry-7d899fbfd7-2j85r"
Oct 30 00:32:00.412102 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.411871    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21e682d8-71fc-4a64-9eda-1b5ed1788fde-registry-certificates\") pod \"image-registry-7d899fbfd7-2j85r\" (UID: \"21e682d8-71fc-4a64-9eda-1b5ed1788fde\") " pod="openshift-image-registry/image-registry-7d899fbfd7-2j85r"
Oct 30 00:32:00.414577 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.412845    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21e682d8-71fc-4a64-9eda-1b5ed1788fde-registry-certificates\") pod \"image-registry-7d899fbfd7-2j85r\" (UID: \"21e682d8-71fc-4a64-9eda-1b5ed1788fde\") " pod="openshift-image-registry/image-registry-7d899fbfd7-2j85r"
Oct 30 00:32:00.414577 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.413950    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21e682d8-71fc-4a64-9eda-1b5ed1788fde-trusted-ca\") pod \"image-registry-7d899fbfd7-2j85r\" (UID: \"21e682d8-71fc-4a64-9eda-1b5ed1788fde\") " pod="openshift-image-registry/image-registry-7d899fbfd7-2j85r"
Oct 30 00:32:00.414577 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.414511    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21e682d8-71fc-4a64-9eda-1b5ed1788fde-ca-trust-extracted\") pod \"image-registry-7d899fbfd7-2j85r\" (UID: \"21e682d8-71fc-4a64-9eda-1b5ed1788fde\") " pod="openshift-image-registry/image-registry-7d899fbfd7-2j85r"
Oct 30 00:32:00.419519 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.419467    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21e682d8-71fc-4a64-9eda-1b5ed1788fde-installation-pull-secrets\") pod \"image-registry-7d899fbfd7-2j85r\" (UID: \"21e682d8-71fc-4a64-9eda-1b5ed1788fde\") " pod="openshift-image-registry/image-registry-7d899fbfd7-2j85r"
Oct 30 00:32:00.424152 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.424122    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21e682d8-71fc-4a64-9eda-1b5ed1788fde-image-registry-private-configuration\") pod \"image-registry-7d899fbfd7-2j85r\" (UID: \"21e682d8-71fc-4a64-9eda-1b5ed1788fde\") " pod="openshift-image-registry/image-registry-7d899fbfd7-2j85r"
Oct 30 00:32:00.424345 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.424304    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21e682d8-71fc-4a64-9eda-1b5ed1788fde-registry-tls\") pod \"image-registry-7d899fbfd7-2j85r\" (UID: \"21e682d8-71fc-4a64-9eda-1b5ed1788fde\") " pod="openshift-image-registry/image-registry-7d899fbfd7-2j85r"
Oct 30 00:32:00.437110 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.437036    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21e682d8-71fc-4a64-9eda-1b5ed1788fde-bound-sa-token\") pod \"image-registry-7d899fbfd7-2j85r\" (UID: \"21e682d8-71fc-4a64-9eda-1b5ed1788fde\") " pod="openshift-image-registry/image-registry-7d899fbfd7-2j85r"
Oct 30 00:32:00.438373 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.438328    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n2hf\" (UniqueName: \"kubernetes.io/projected/21e682d8-71fc-4a64-9eda-1b5ed1788fde-kube-api-access-8n2hf\") pod \"image-registry-7d899fbfd7-2j85r\" (UID: \"21e682d8-71fc-4a64-9eda-1b5ed1788fde\") " pod="openshift-image-registry/image-registry-7d899fbfd7-2j85r"
Oct 30 00:32:00.460082 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.460061    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d899fbfd7-2j85r"
Oct 30 00:32:00.643677 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.643643    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-p5zpd"]
Oct 30 00:32:00.646998 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:32:00.646973    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7258c681_0692_4203_b3da_a3bab6b298dd.slice/crio-085d696dd2d6f3e8b921e0e375a1e859b06cc5f0103f12e96a549367774a2e98 WatchSource:0}: Error finding container 085d696dd2d6f3e8b921e0e375a1e859b06cc5f0103f12e96a549367774a2e98: Status 404 returned error can't find the container with id 085d696dd2d6f3e8b921e0e375a1e859b06cc5f0103f12e96a549367774a2e98
Oct 30 00:32:00.680252 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.680221    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-r7bf7"]
Oct 30 00:32:00.683234 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:32:00.683206    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a3d031e_b63e_4201_afb9_ec659b67ac1e.slice/crio-c7008aa657a66fff14d84bb68476e7651181869567a0c7615c0a64438345c767 WatchSource:0}: Error finding container c7008aa657a66fff14d84bb68476e7651181869567a0c7615c0a64438345c767: Status 404 returned error can't find the container with id c7008aa657a66fff14d84bb68476e7651181869567a0c7615c0a64438345c767
Oct 30 00:32:00.692708 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.692605    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cc457646f-sjj89"]
Oct 30 00:32:00.695250 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:32:00.695224    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72a65fa9_4477_434a_aaac_0394505927e5.slice/crio-be1211d9c46ab1f26d76df43bb0e4726e0a8f5eecc90f2bb8208974a0a0da7fe WatchSource:0}: Error finding container be1211d9c46ab1f26d76df43bb0e4726e0a8f5eecc90f2bb8208974a0a0da7fe: Status 404 returned error can't find the container with id be1211d9c46ab1f26d76df43bb0e4726e0a8f5eecc90f2bb8208974a0a0da7fe
Oct 30 00:32:00.699896 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.699879    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pxfhp"
Oct 30 00:32:00.757765 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.757735    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d899fbfd7-2j85r"]
Oct 30 00:32:00.761112 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:32:00.761077    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21e682d8_71fc_4a64_9eda_1b5ed1788fde.slice/crio-38f6e53f77f98e3810f9f43aac709d567d0ba5a928e344bfacf284f1c0efdb24 WatchSource:0}: Error finding container 38f6e53f77f98e3810f9f43aac709d567d0ba5a928e344bfacf284f1c0efdb24: Status 404 returned error can't find the container with id 38f6e53f77f98e3810f9f43aac709d567d0ba5a928e344bfacf284f1c0efdb24
Oct 30 00:32:00.806172 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.806145    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-p5zpd" event={"ID":"7258c681-0692-4203-b3da-a3bab6b298dd","Type":"ContainerStarted","Data":"085d696dd2d6f3e8b921e0e375a1e859b06cc5f0103f12e96a549367774a2e98"}
Oct 30 00:32:00.807644 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.807619    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-r7bf7" event={"ID":"4a3d031e-b63e-4201-afb9-ec659b67ac1e","Type":"ContainerStarted","Data":"c7008aa657a66fff14d84bb68476e7651181869567a0c7615c0a64438345c767"}
Oct 30 00:32:00.809020 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.808994    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d899fbfd7-2j85r" event={"ID":"21e682d8-71fc-4a64-9eda-1b5ed1788fde","Type":"ContainerStarted","Data":"38f6e53f77f98e3810f9f43aac709d567d0ba5a928e344bfacf284f1c0efdb24"}
Oct 30 00:32:00.810618 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:00.810596    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cc457646f-sjj89" event={"ID":"72a65fa9-4477-434a-aaac-0394505927e5","Type":"ContainerStarted","Data":"be1211d9c46ab1f26d76df43bb0e4726e0a8f5eecc90f2bb8208974a0a0da7fe"}
Oct 30 00:32:01.818004 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:01.817964    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d899fbfd7-2j85r" event={"ID":"21e682d8-71fc-4a64-9eda-1b5ed1788fde","Type":"ContainerStarted","Data":"6cc9a128ef017ea746c5db4d7fb4e47b9396553048d82e89cfd8ab406b8a6f8d"}
Oct 30 00:32:01.818684 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:01.818664    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7d899fbfd7-2j85r"
Oct 30 00:32:01.863369 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:01.863313    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7d899fbfd7-2j85r" podStartSLOduration=1.863295 podStartE2EDuration="1.863295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:01.862068445 +0000 UTC m=+12.249744834" watchObservedRunningTime="2026-10-30 00:32:01.863295 +0000 UTC m=+12.250971392"
Oct 30 00:32:02.825285 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:02.825246    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-p5zpd" event={"ID":"7258c681-0692-4203-b3da-a3bab6b298dd","Type":"ContainerStarted","Data":"5fc6b565c37d3093cb9b460497cd8fd40d2124ddaa6e89ec2c8031fd5874213d"}
Oct 30 00:32:02.825789 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:02.825609    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-p5zpd"
Oct 30 00:32:02.831545 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:02.831521    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-p5zpd"
Oct 30 00:32:02.863273 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:02.863225    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-p5zpd" podStartSLOduration=2.728340217 podStartE2EDuration="3.863208362s" totalImagesPullingTime="1.134868145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:31:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:02.862923346 +0000 UTC m=+13.250599735" watchObservedRunningTime="2026-10-30 00:32:02.863208362 +0000 UTC m=+13.250884753"
Oct 30 00:32:04.788521 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:04.788491    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9ml2g"
Oct 30 00:32:04.839374 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:04.839326    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cc457646f-sjj89" event={"ID":"72a65fa9-4477-434a-aaac-0394505927e5","Type":"ContainerStarted","Data":"de2caceb2c2f9c010a09716d4a14759468bde844cff71f7d34f33b6bae84745d"}
Oct 30 00:32:08.154453 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.154389    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7cc457646f-sjj89" podStartSLOduration=6.029252169 podStartE2EDuration="9.154372542s" totalImagesPullingTime="3.125120373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:31:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:04.881877798 +0000 UTC m=+15.269554188" watchObservedRunningTime="2026-10-30 00:32:08.154372542 +0000 UTC m=+18.542048930"
Oct 30 00:32:08.155011 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.154576    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-67pwz"]
Oct 30 00:32:08.158113 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.158092    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-67pwz"
Oct 30 00:32:08.182669 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.182639    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 00:32:08.192962 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.192933    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 00:32:08.193155 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.193005    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 00:32:08.193155 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.193097    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 00:32:08.198671 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.198647    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 00:32:08.198842 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.198817    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-w25sk\""
Oct 30 00:32:08.198937 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.198917    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 00:32:08.277590 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.277557    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8e23449e-a558-4347-bed9-e631437e9c73-node-exporter-wtmp\") pod \"node-exporter-67pwz\" (UID: \"8e23449e-a558-4347-bed9-e631437e9c73\") " pod="openshift-monitoring/node-exporter-67pwz"
Oct 30 00:32:08.277590 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.277600    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8e23449e-a558-4347-bed9-e631437e9c73-node-exporter-accelerators-collector-config\") pod \"node-exporter-67pwz\" (UID: \"8e23449e-a558-4347-bed9-e631437e9c73\") " pod="openshift-monitoring/node-exporter-67pwz"
Oct 30 00:32:08.277846 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.277631    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhxgw\" (UniqueName: \"kubernetes.io/projected/8e23449e-a558-4347-bed9-e631437e9c73-kube-api-access-dhxgw\") pod \"node-exporter-67pwz\" (UID: \"8e23449e-a558-4347-bed9-e631437e9c73\") " pod="openshift-monitoring/node-exporter-67pwz"
Oct 30 00:32:08.277846 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.277733    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e23449e-a558-4347-bed9-e631437e9c73-metrics-client-ca\") pod \"node-exporter-67pwz\" (UID: \"8e23449e-a558-4347-bed9-e631437e9c73\") " pod="openshift-monitoring/node-exporter-67pwz"
Oct 30 00:32:08.277846 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.277826    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8e23449e-a558-4347-bed9-e631437e9c73-node-exporter-tls\") pod \"node-exporter-67pwz\" (UID: \"8e23449e-a558-4347-bed9-e631437e9c73\") " pod="openshift-monitoring/node-exporter-67pwz"
Oct 30 00:32:08.277986 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.277856    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e23449e-a558-4347-bed9-e631437e9c73-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-67pwz\" (UID: \"8e23449e-a558-4347-bed9-e631437e9c73\") " pod="openshift-monitoring/node-exporter-67pwz"
Oct 30 00:32:08.277986 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.277877    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e23449e-a558-4347-bed9-e631437e9c73-sys\") pod \"node-exporter-67pwz\" (UID: \"8e23449e-a558-4347-bed9-e631437e9c73\") " pod="openshift-monitoring/node-exporter-67pwz"
Oct 30 00:32:08.277986 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.277890    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8e23449e-a558-4347-bed9-e631437e9c73-root\") pod \"node-exporter-67pwz\" (UID: \"8e23449e-a558-4347-bed9-e631437e9c73\") " pod="openshift-monitoring/node-exporter-67pwz"
Oct 30 00:32:08.277986 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.277959    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8e23449e-a558-4347-bed9-e631437e9c73-node-exporter-textfile\") pod \"node-exporter-67pwz\" (UID: \"8e23449e-a558-4347-bed9-e631437e9c73\") " pod="openshift-monitoring/node-exporter-67pwz"
Oct 30 00:32:08.379377 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.379325    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8e23449e-a558-4347-bed9-e631437e9c73-node-exporter-tls\") pod \"node-exporter-67pwz\" (UID: \"8e23449e-a558-4347-bed9-e631437e9c73\") " pod="openshift-monitoring/node-exporter-67pwz"
Oct 30 00:32:08.379564 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.379397    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e23449e-a558-4347-bed9-e631437e9c73-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-67pwz\" (UID: \"8e23449e-a558-4347-bed9-e631437e9c73\") " pod="openshift-monitoring/node-exporter-67pwz"
Oct 30 00:32:08.379564 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.379484    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e23449e-a558-4347-bed9-e631437e9c73-sys\") pod \"node-exporter-67pwz\" (UID: \"8e23449e-a558-4347-bed9-e631437e9c73\") " pod="openshift-monitoring/node-exporter-67pwz"
Oct 30 00:32:08.379564 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.379521    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e23449e-a558-4347-bed9-e631437e9c73-sys\") pod \"node-exporter-67pwz\" (UID: \"8e23449e-a558-4347-bed9-e631437e9c73\") " pod="openshift-monitoring/node-exporter-67pwz"
Oct 30 00:32:08.379564 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.379536    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8e23449e-a558-4347-bed9-e631437e9c73-root\") pod \"node-exporter-67pwz\" (UID: \"8e23449e-a558-4347-bed9-e631437e9c73\") " pod="openshift-monitoring/node-exporter-67pwz"
Oct 30 00:32:08.379564 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.379560    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8e23449e-a558-4347-bed9-e631437e9c73-node-exporter-textfile\") pod \"node-exporter-67pwz\" (UID: \"8e23449e-a558-4347-bed9-e631437e9c73\") " pod="openshift-monitoring/node-exporter-67pwz"
Oct 30 00:32:08.379716 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.379595    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8e23449e-a558-4347-bed9-e631437e9c73-node-exporter-wtmp\") pod \"node-exporter-67pwz\" (UID: \"8e23449e-a558-4347-bed9-e631437e9c73\") " pod="openshift-monitoring/node-exporter-67pwz"
Oct 30 00:32:08.379716 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.379613    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8e23449e-a558-4347-bed9-e631437e9c73-node-exporter-accelerators-collector-config\") pod \"node-exporter-67pwz\" (UID: \"8e23449e-a558-4347-bed9-e631437e9c73\") " pod="openshift-monitoring/node-exporter-67pwz"
Oct 30 00:32:08.379716 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.379636    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhxgw\" (UniqueName: \"kubernetes.io/projected/8e23449e-a558-4347-bed9-e631437e9c73-kube-api-access-dhxgw\") pod \"node-exporter-67pwz\" (UID: \"8e23449e-a558-4347-bed9-e631437e9c73\") " pod="openshift-monitoring/node-exporter-67pwz"
Oct 30 00:32:08.379716 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.379661    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e23449e-a558-4347-bed9-e631437e9c73-metrics-client-ca\") pod \"node-exporter-67pwz\" (UID: \"8e23449e-a558-4347-bed9-e631437e9c73\") " pod="openshift-monitoring/node-exporter-67pwz"
Oct 30 00:32:08.379965 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.379908    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8e23449e-a558-4347-bed9-e631437e9c73-root\") pod \"node-exporter-67pwz\" (UID: \"8e23449e-a558-4347-bed9-e631437e9c73\") " pod="openshift-monitoring/node-exporter-67pwz"
Oct 30 00:32:08.380077 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.380014    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8e23449e-a558-4347-bed9-e631437e9c73-node-exporter-wtmp\") pod \"node-exporter-67pwz\" (UID: \"8e23449e-a558-4347-bed9-e631437e9c73\") " pod="openshift-monitoring/node-exporter-67pwz"
Oct 30 00:32:08.380137 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.380107    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8e23449e-a558-4347-bed9-e631437e9c73-node-exporter-textfile\") pod \"node-exporter-67pwz\" (UID: \"8e23449e-a558-4347-bed9-e631437e9c73\") " pod="openshift-monitoring/node-exporter-67pwz"
Oct 30 00:32:08.380337 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.380314    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e23449e-a558-4347-bed9-e631437e9c73-metrics-client-ca\") pod \"node-exporter-67pwz\" (UID: \"8e23449e-a558-4347-bed9-e631437e9c73\") " pod="openshift-monitoring/node-exporter-67pwz"
Oct 30 00:32:08.380761 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.380736    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8e23449e-a558-4347-bed9-e631437e9c73-node-exporter-accelerators-collector-config\") pod \"node-exporter-67pwz\" (UID: \"8e23449e-a558-4347-bed9-e631437e9c73\") " pod="openshift-monitoring/node-exporter-67pwz"
Oct 30 00:32:08.382413 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.382385    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8e23449e-a558-4347-bed9-e631437e9c73-node-exporter-tls\") pod \"node-exporter-67pwz\" (UID: \"8e23449e-a558-4347-bed9-e631437e9c73\") " pod="openshift-monitoring/node-exporter-67pwz"
Oct 30 00:32:08.382710 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.382690    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8e23449e-a558-4347-bed9-e631437e9c73-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-67pwz\" (UID: \"8e23449e-a558-4347-bed9-e631437e9c73\") " pod="openshift-monitoring/node-exporter-67pwz"
Oct 30 00:32:08.402992 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.402957    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhxgw\" (UniqueName: \"kubernetes.io/projected/8e23449e-a558-4347-bed9-e631437e9c73-kube-api-access-dhxgw\") pod \"node-exporter-67pwz\" (UID: \"8e23449e-a558-4347-bed9-e631437e9c73\") " pod="openshift-monitoring/node-exporter-67pwz"
Oct 30 00:32:08.473091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.473000    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-67pwz"
Oct 30 00:32:08.484259 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:32:08.484218    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e23449e_a558_4347_bed9_e631437e9c73.slice/crio-f2df8c469efdae9d4131576f5b6053bd87058df178b54b0975f09154458f0846 WatchSource:0}: Error finding container f2df8c469efdae9d4131576f5b6053bd87058df178b54b0975f09154458f0846: Status 404 returned error can't find the container with id f2df8c469efdae9d4131576f5b6053bd87058df178b54b0975f09154458f0846
Oct 30 00:32:08.856131 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:08.855858    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-67pwz" event={"ID":"8e23449e-a558-4347-bed9-e631437e9c73","Type":"ContainerStarted","Data":"f2df8c469efdae9d4131576f5b6053bd87058df178b54b0975f09154458f0846"}
Oct 30 00:32:09.861226 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:09.861132    4393 generic.go:356] "Generic (PLEG): container finished" podID="8e23449e-a558-4347-bed9-e631437e9c73" containerID="8fdb267754fa02109a069e2a1617cae229ab25463d7284717b667a20163c1111" exitCode=0
Oct 30 00:32:09.861226 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:09.861211    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-67pwz" event={"ID":"8e23449e-a558-4347-bed9-e631437e9c73","Type":"ContainerDied","Data":"8fdb267754fa02109a069e2a1617cae229ab25463d7284717b667a20163c1111"}
Oct 30 00:32:10.358738 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.358702    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7cc457646f-sjj89"
Oct 30 00:32:10.358981 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.358768    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7cc457646f-sjj89"
Oct 30 00:32:10.365219 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.365190    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7cc457646f-sjj89"
Oct 30 00:32:10.421683 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.421645    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-648d8c566c-nw7rf"]
Oct 30 00:32:10.424592 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.424556    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf"
Oct 30 00:32:10.441816 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.441730    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 00:32:10.441816 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.441750    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 00:32:10.442079 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.441838    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 00:32:10.442079 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.441867    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 00:32:10.442079 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.441938    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-t2lq7\""
Oct 30 00:32:10.442079 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.442010    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 00:32:10.442343 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.442323    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8pss259pemroa\""
Oct 30 00:32:10.447345 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.447318    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-648d8c566c-nw7rf"]
Oct 30 00:32:10.600061 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.600020    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4af88d81-38c0-4edf-be60-d8c27b330779-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-648d8c566c-nw7rf\" (UID: \"4af88d81-38c0-4edf-be60-d8c27b330779\") " pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf"
Oct 30 00:32:10.600061 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.600065    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4af88d81-38c0-4edf-be60-d8c27b330779-metrics-client-ca\") pod \"thanos-querier-648d8c566c-nw7rf\" (UID: \"4af88d81-38c0-4edf-be60-d8c27b330779\") " pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf"
Oct 30 00:32:10.600330 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.600196    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4af88d81-38c0-4edf-be60-d8c27b330779-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-648d8c566c-nw7rf\" (UID: \"4af88d81-38c0-4edf-be60-d8c27b330779\") " pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf"
Oct 30 00:32:10.600330 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.600243    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4af88d81-38c0-4edf-be60-d8c27b330779-tmp\") pod \"thanos-querier-648d8c566c-nw7rf\" (UID: \"4af88d81-38c0-4edf-be60-d8c27b330779\") " pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf"
Oct 30 00:32:10.600330 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.600314    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4af88d81-38c0-4edf-be60-d8c27b330779-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-648d8c566c-nw7rf\" (UID: \"4af88d81-38c0-4edf-be60-d8c27b330779\") " pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf"
Oct 30 00:32:10.600481 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.600373    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4af88d81-38c0-4edf-be60-d8c27b330779-secret-thanos-querier-tls\") pod \"thanos-querier-648d8c566c-nw7rf\" (UID: \"4af88d81-38c0-4edf-be60-d8c27b330779\") " pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf"
Oct 30 00:32:10.600481 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.600392    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4af88d81-38c0-4edf-be60-d8c27b330779-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-648d8c566c-nw7rf\" (UID: \"4af88d81-38c0-4edf-be60-d8c27b330779\") " pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf"
Oct 30 00:32:10.600575 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.600487    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vml5n\" (UniqueName: \"kubernetes.io/projected/4af88d81-38c0-4edf-be60-d8c27b330779-kube-api-access-vml5n\") pod \"thanos-querier-648d8c566c-nw7rf\" (UID: \"4af88d81-38c0-4edf-be60-d8c27b330779\") " pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf"
Oct 30 00:32:10.600575 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.600521    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4af88d81-38c0-4edf-be60-d8c27b330779-secret-grpc-tls\") pod \"thanos-querier-648d8c566c-nw7rf\" (UID: \"4af88d81-38c0-4edf-be60-d8c27b330779\") " pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf"
Oct 30 00:32:10.701305 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.701261    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4af88d81-38c0-4edf-be60-d8c27b330779-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-648d8c566c-nw7rf\" (UID: \"4af88d81-38c0-4edf-be60-d8c27b330779\") " pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf"
Oct 30 00:32:10.701305 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.701308    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4af88d81-38c0-4edf-be60-d8c27b330779-metrics-client-ca\") pod \"thanos-querier-648d8c566c-nw7rf\" (UID: \"4af88d81-38c0-4edf-be60-d8c27b330779\") " pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf"
Oct 30 00:32:10.701561 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.701430    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4af88d81-38c0-4edf-be60-d8c27b330779-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-648d8c566c-nw7rf\" (UID: \"4af88d81-38c0-4edf-be60-d8c27b330779\") " pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf"
Oct 30 00:32:10.701561 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.701474    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4af88d81-38c0-4edf-be60-d8c27b330779-tmp\") pod \"thanos-querier-648d8c566c-nw7rf\" (UID: \"4af88d81-38c0-4edf-be60-d8c27b330779\") " pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf"
Oct 30 00:32:10.701561 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.701545    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4af88d81-38c0-4edf-be60-d8c27b330779-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-648d8c566c-nw7rf\" (UID: \"4af88d81-38c0-4edf-be60-d8c27b330779\") " pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf"
Oct 30 00:32:10.701696 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.701598    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4af88d81-38c0-4edf-be60-d8c27b330779-secret-thanos-querier-tls\") pod \"thanos-querier-648d8c566c-nw7rf\" (UID: \"4af88d81-38c0-4edf-be60-d8c27b330779\") " pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf"
Oct 30 00:32:10.701745 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.701684    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4af88d81-38c0-4edf-be60-d8c27b330779-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-648d8c566c-nw7rf\" (UID: \"4af88d81-38c0-4edf-be60-d8c27b330779\") " pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf"
Oct 30 00:32:10.701822 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.701743    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vml5n\" (UniqueName: \"kubernetes.io/projected/4af88d81-38c0-4edf-be60-d8c27b330779-kube-api-access-vml5n\") pod \"thanos-querier-648d8c566c-nw7rf\" (UID: \"4af88d81-38c0-4edf-be60-d8c27b330779\") " pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf"
Oct 30 00:32:10.701822 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.701809    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4af88d81-38c0-4edf-be60-d8c27b330779-secret-grpc-tls\") pod \"thanos-querier-648d8c566c-nw7rf\" (UID: \"4af88d81-38c0-4edf-be60-d8c27b330779\") " pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf"
Oct 30 00:32:10.702914 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.702806    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4af88d81-38c0-4edf-be60-d8c27b330779-metrics-client-ca\") pod \"thanos-querier-648d8c566c-nw7rf\" (UID: \"4af88d81-38c0-4edf-be60-d8c27b330779\") " pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf"
Oct 30 00:32:10.705000 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.704959    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4af88d81-38c0-4edf-be60-d8c27b330779-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-648d8c566c-nw7rf\" (UID: \"4af88d81-38c0-4edf-be60-d8c27b330779\") " pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf"
Oct 30 00:32:10.709927 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.705232    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4af88d81-38c0-4edf-be60-d8c27b330779-tmp\") pod \"thanos-querier-648d8c566c-nw7rf\" (UID: \"4af88d81-38c0-4edf-be60-d8c27b330779\") " pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf"
Oct 30 00:32:10.709927 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.705519    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4af88d81-38c0-4edf-be60-d8c27b330779-secret-thanos-querier-tls\") pod \"thanos-querier-648d8c566c-nw7rf\" (UID: \"4af88d81-38c0-4edf-be60-d8c27b330779\") " pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf"
Oct 30 00:32:10.709927 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.705856    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4af88d81-38c0-4edf-be60-d8c27b330779-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-648d8c566c-nw7rf\" (UID: \"4af88d81-38c0-4edf-be60-d8c27b330779\") " pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf"
Oct 30 00:32:10.709927 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.706156    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4af88d81-38c0-4edf-be60-d8c27b330779-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-648d8c566c-nw7rf\" (UID: \"4af88d81-38c0-4edf-be60-d8c27b330779\") " pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf"
Oct 30 00:32:10.709927 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.706261    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4af88d81-38c0-4edf-be60-d8c27b330779-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-648d8c566c-nw7rf\" (UID: \"4af88d81-38c0-4edf-be60-d8c27b330779\") " pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf"
Oct 30 00:32:10.709927 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.708065    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4af88d81-38c0-4edf-be60-d8c27b330779-secret-grpc-tls\") pod \"thanos-querier-648d8c566c-nw7rf\" (UID: \"4af88d81-38c0-4edf-be60-d8c27b330779\") " pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf"
Oct 30 00:32:10.729604 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.729568    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vml5n\" (UniqueName: \"kubernetes.io/projected/4af88d81-38c0-4edf-be60-d8c27b330779-kube-api-access-vml5n\") pod \"thanos-querier-648d8c566c-nw7rf\" (UID: \"4af88d81-38c0-4edf-be60-d8c27b330779\") " pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf"
Oct 30 00:32:10.743276 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.743240    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf"
Oct 30 00:32:10.869044 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.869004    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-67pwz" event={"ID":"8e23449e-a558-4347-bed9-e631437e9c73","Type":"ContainerStarted","Data":"8e15d6d8445924ed8c849687375c84c4968c4bc30e996acf58404547bbaee441"}
Oct 30 00:32:10.869970 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.869100    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-67pwz" event={"ID":"8e23449e-a558-4347-bed9-e631437e9c73","Type":"ContainerStarted","Data":"99f36f81b94ea9cd40fed0572afdff99733633834c1bf5b56680cffb9f0f522a"}
Oct 30 00:32:10.874433 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.874405    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7cc457646f-sjj89"
Oct 30 00:32:10.922516 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:10.922460    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-67pwz" podStartSLOduration=1.839256507 podStartE2EDuration="2.922445222s" totalImagesPullingTime="995.578167ms" totalInitContainerRuntime="87.610548ms" isStatefulPod=true podCreationTimestamp="2026-10-30 00:32:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:10.92164439 +0000 UTC m=+21.309320783" watchObservedRunningTime="2026-10-30 00:32:10.922445222 +0000 UTC m=+21.310121611"
Oct 30 00:32:11.067184 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:11.067091    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-648d8c566c-nw7rf"]
Oct 30 00:32:11.070793 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:32:11.070736    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af88d81_38c0_4edf_be60_d8c27b330779.slice/crio-6e30dc8e7028db2764fa4154e2867e85729822d0f6a1c8b440d52a65bc2794de WatchSource:0}: Error finding container 6e30dc8e7028db2764fa4154e2867e85729822d0f6a1c8b440d52a65bc2794de: Status 404 returned error can't find the container with id 6e30dc8e7028db2764fa4154e2867e85729822d0f6a1c8b440d52a65bc2794de
Oct 30 00:32:11.874312 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:11.874270    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf" event={"ID":"4af88d81-38c0-4edf-be60-d8c27b330779","Type":"ContainerStarted","Data":"6e30dc8e7028db2764fa4154e2867e85729822d0f6a1c8b440d52a65bc2794de"}
Oct 30 00:32:13.470591 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.470558    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-766c9f88f-jvcvh"]
Oct 30 00:32:13.473315 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.473295    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-766c9f88f-jvcvh"
Oct 30 00:32:13.498522 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.498411    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-766c9f88f-jvcvh"]
Oct 30 00:32:13.509204 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.509170    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 00:32:13.509204 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.509177    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 00:32:13.509447 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.509223    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 00:32:13.509447 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.509334    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-hjd8x\""
Oct 30 00:32:13.509711 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.509678    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 00:32:13.521654 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.521618    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 00:32:13.522079 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.522058    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 00:32:13.556626 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.556587    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6wzd9"
Oct 30 00:32:13.632510 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.632470    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2fa1bcf9-0df3-4978-a8a4-493c6b066681-metrics-client-ca\") pod \"telemeter-client-766c9f88f-jvcvh\" (UID: \"2fa1bcf9-0df3-4978-a8a4-493c6b066681\") " pod="openshift-monitoring/telemeter-client-766c9f88f-jvcvh"
Oct 30 00:32:13.632717 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.632525    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fa1bcf9-0df3-4978-a8a4-493c6b066681-telemeter-trusted-ca-bundle\") pod \"telemeter-client-766c9f88f-jvcvh\" (UID: \"2fa1bcf9-0df3-4978-a8a4-493c6b066681\") " pod="openshift-monitoring/telemeter-client-766c9f88f-jvcvh"
Oct 30 00:32:13.632717 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.632643    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2fa1bcf9-0df3-4978-a8a4-493c6b066681-federate-client-tls\") pod \"telemeter-client-766c9f88f-jvcvh\" (UID: \"2fa1bcf9-0df3-4978-a8a4-493c6b066681\") " pod="openshift-monitoring/telemeter-client-766c9f88f-jvcvh"
Oct 30 00:32:13.632717 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.632696    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fa1bcf9-0df3-4978-a8a4-493c6b066681-serving-certs-ca-bundle\") pod \"telemeter-client-766c9f88f-jvcvh\" (UID: \"2fa1bcf9-0df3-4978-a8a4-493c6b066681\") " pod="openshift-monitoring/telemeter-client-766c9f88f-jvcvh"
Oct 30 00:32:13.632952 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.632888    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2fa1bcf9-0df3-4978-a8a4-493c6b066681-telemeter-client-tls\") pod \"telemeter-client-766c9f88f-jvcvh\" (UID: \"2fa1bcf9-0df3-4978-a8a4-493c6b066681\") " pod="openshift-monitoring/telemeter-client-766c9f88f-jvcvh"
Oct 30 00:32:13.632952 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.632929    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2fa1bcf9-0df3-4978-a8a4-493c6b066681-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-766c9f88f-jvcvh\" (UID: \"2fa1bcf9-0df3-4978-a8a4-493c6b066681\") " pod="openshift-monitoring/telemeter-client-766c9f88f-jvcvh"
Oct 30 00:32:13.633048 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.633028    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2fa1bcf9-0df3-4978-a8a4-493c6b066681-secret-telemeter-client\") pod \"telemeter-client-766c9f88f-jvcvh\" (UID: \"2fa1bcf9-0df3-4978-a8a4-493c6b066681\") " pod="openshift-monitoring/telemeter-client-766c9f88f-jvcvh"
Oct 30 00:32:13.633090 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.633067    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z794m\" (UniqueName: \"kubernetes.io/projected/2fa1bcf9-0df3-4978-a8a4-493c6b066681-kube-api-access-z794m\") pod \"telemeter-client-766c9f88f-jvcvh\" (UID: \"2fa1bcf9-0df3-4978-a8a4-493c6b066681\") " pod="openshift-monitoring/telemeter-client-766c9f88f-jvcvh"
Oct 30 00:32:13.734141 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.734094    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2fa1bcf9-0df3-4978-a8a4-493c6b066681-telemeter-client-tls\") pod \"telemeter-client-766c9f88f-jvcvh\" (UID: \"2fa1bcf9-0df3-4978-a8a4-493c6b066681\") " pod="openshift-monitoring/telemeter-client-766c9f88f-jvcvh"
Oct 30 00:32:13.734141 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.734146    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2fa1bcf9-0df3-4978-a8a4-493c6b066681-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-766c9f88f-jvcvh\" (UID: \"2fa1bcf9-0df3-4978-a8a4-493c6b066681\") " pod="openshift-monitoring/telemeter-client-766c9f88f-jvcvh"
Oct 30 00:32:13.734397 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.734198    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2fa1bcf9-0df3-4978-a8a4-493c6b066681-secret-telemeter-client\") pod \"telemeter-client-766c9f88f-jvcvh\" (UID: \"2fa1bcf9-0df3-4978-a8a4-493c6b066681\") " pod="openshift-monitoring/telemeter-client-766c9f88f-jvcvh"
Oct 30 00:32:13.734397 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.734232    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z794m\" (UniqueName: \"kubernetes.io/projected/2fa1bcf9-0df3-4978-a8a4-493c6b066681-kube-api-access-z794m\") pod \"telemeter-client-766c9f88f-jvcvh\" (UID: \"2fa1bcf9-0df3-4978-a8a4-493c6b066681\") " pod="openshift-monitoring/telemeter-client-766c9f88f-jvcvh"
Oct 30 00:32:13.734397 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.734279    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2fa1bcf9-0df3-4978-a8a4-493c6b066681-metrics-client-ca\") pod \"telemeter-client-766c9f88f-jvcvh\" (UID: \"2fa1bcf9-0df3-4978-a8a4-493c6b066681\") " pod="openshift-monitoring/telemeter-client-766c9f88f-jvcvh"
Oct 30 00:32:13.734397 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.734306    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fa1bcf9-0df3-4978-a8a4-493c6b066681-telemeter-trusted-ca-bundle\") pod \"telemeter-client-766c9f88f-jvcvh\" (UID: \"2fa1bcf9-0df3-4978-a8a4-493c6b066681\") " pod="openshift-monitoring/telemeter-client-766c9f88f-jvcvh"
Oct 30 00:32:13.734397 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.734354    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2fa1bcf9-0df3-4978-a8a4-493c6b066681-federate-client-tls\") pod \"telemeter-client-766c9f88f-jvcvh\" (UID: \"2fa1bcf9-0df3-4978-a8a4-493c6b066681\") " pod="openshift-monitoring/telemeter-client-766c9f88f-jvcvh"
Oct 30 00:32:13.734397 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.734377    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fa1bcf9-0df3-4978-a8a4-493c6b066681-serving-certs-ca-bundle\") pod \"telemeter-client-766c9f88f-jvcvh\" (UID: \"2fa1bcf9-0df3-4978-a8a4-493c6b066681\") " pod="openshift-monitoring/telemeter-client-766c9f88f-jvcvh"
Oct 30 00:32:13.735697 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.735331    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fa1bcf9-0df3-4978-a8a4-493c6b066681-serving-certs-ca-bundle\") pod \"telemeter-client-766c9f88f-jvcvh\" (UID: \"2fa1bcf9-0df3-4978-a8a4-493c6b066681\") " pod="openshift-monitoring/telemeter-client-766c9f88f-jvcvh"
Oct 30 00:32:13.735697 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.735346    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2fa1bcf9-0df3-4978-a8a4-493c6b066681-metrics-client-ca\") pod \"telemeter-client-766c9f88f-jvcvh\" (UID: \"2fa1bcf9-0df3-4978-a8a4-493c6b066681\") " pod="openshift-monitoring/telemeter-client-766c9f88f-jvcvh"
Oct 30 00:32:13.736688 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.736658    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fa1bcf9-0df3-4978-a8a4-493c6b066681-telemeter-trusted-ca-bundle\") pod \"telemeter-client-766c9f88f-jvcvh\" (UID: \"2fa1bcf9-0df3-4978-a8a4-493c6b066681\") " pod="openshift-monitoring/telemeter-client-766c9f88f-jvcvh"
Oct 30 00:32:13.737314 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.737285    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2fa1bcf9-0df3-4978-a8a4-493c6b066681-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-766c9f88f-jvcvh\" (UID: \"2fa1bcf9-0df3-4978-a8a4-493c6b066681\") " pod="openshift-monitoring/telemeter-client-766c9f88f-jvcvh"
Oct 30 00:32:13.737983 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.737936    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2fa1bcf9-0df3-4978-a8a4-493c6b066681-telemeter-client-tls\") pod \"telemeter-client-766c9f88f-jvcvh\" (UID: \"2fa1bcf9-0df3-4978-a8a4-493c6b066681\") " pod="openshift-monitoring/telemeter-client-766c9f88f-jvcvh"
Oct 30 00:32:13.739427 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.739407    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2fa1bcf9-0df3-4978-a8a4-493c6b066681-secret-telemeter-client\") pod \"telemeter-client-766c9f88f-jvcvh\" (UID: \"2fa1bcf9-0df3-4978-a8a4-493c6b066681\") " pod="openshift-monitoring/telemeter-client-766c9f88f-jvcvh"
Oct 30 00:32:13.739660 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.739636    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2fa1bcf9-0df3-4978-a8a4-493c6b066681-federate-client-tls\") pod \"telemeter-client-766c9f88f-jvcvh\" (UID: \"2fa1bcf9-0df3-4978-a8a4-493c6b066681\") " pod="openshift-monitoring/telemeter-client-766c9f88f-jvcvh"
Oct 30 00:32:13.759180 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.759111    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z794m\" (UniqueName: \"kubernetes.io/projected/2fa1bcf9-0df3-4978-a8a4-493c6b066681-kube-api-access-z794m\") pod \"telemeter-client-766c9f88f-jvcvh\" (UID: \"2fa1bcf9-0df3-4978-a8a4-493c6b066681\") " pod="openshift-monitoring/telemeter-client-766c9f88f-jvcvh"
Oct 30 00:32:13.788638 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.788597    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-766c9f88f-jvcvh"
Oct 30 00:32:13.885031 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.884992    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf" event={"ID":"4af88d81-38c0-4edf-be60-d8c27b330779","Type":"ContainerStarted","Data":"d88bcc3ad7c31a453a471c9ccdbd76aaa0e7b5089f73d7dc45ce3c3078d89af5"}
Oct 30 00:32:13.885221 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.885078    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf" event={"ID":"4af88d81-38c0-4edf-be60-d8c27b330779","Type":"ContainerStarted","Data":"6db976691b547fb0af5d78a1a64f1df1961e7d514eb94ee57a57a2eb5931e279"}
Oct 30 00:32:13.885221 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:13.885115    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf" event={"ID":"4af88d81-38c0-4edf-be60-d8c27b330779","Type":"ContainerStarted","Data":"0d0372b3ee24b139a53cb4dacd81b2ed7b951c380b064b5905fe65d1962b8deb"}
Oct 30 00:32:14.126809 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:14.126743    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-766c9f88f-jvcvh"]
Oct 30 00:32:14.130885 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:32:14.130853    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa1bcf9_0df3_4978_a8a4_493c6b066681.slice/crio-c237c22f89a1b8eb8b648e4e0cedcf6147d29d47a5311c06c60638c218debccf WatchSource:0}: Error finding container c237c22f89a1b8eb8b648e4e0cedcf6147d29d47a5311c06c60638c218debccf: Status 404 returned error can't find the container with id c237c22f89a1b8eb8b648e4e0cedcf6147d29d47a5311c06c60638c218debccf
Oct 30 00:32:14.890860 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:14.890812    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-766c9f88f-jvcvh" event={"ID":"2fa1bcf9-0df3-4978-a8a4-493c6b066681","Type":"ContainerStarted","Data":"c237c22f89a1b8eb8b648e4e0cedcf6147d29d47a5311c06c60638c218debccf"}
Oct 30 00:32:14.894727 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:14.894691    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf" event={"ID":"4af88d81-38c0-4edf-be60-d8c27b330779","Type":"ContainerStarted","Data":"8ea67de0a9a90669c2814e1503d3fa570a935d901208323df6cfbc68a2b96294"}
Oct 30 00:32:14.894904 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:14.894765    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf" event={"ID":"4af88d81-38c0-4edf-be60-d8c27b330779","Type":"ContainerStarted","Data":"4e59b59d4f25e6e5b258ba4c3aae293cb4ae4535090e983af81dce137427919f"}
Oct 30 00:32:14.894904 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:14.894814    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf" event={"ID":"4af88d81-38c0-4edf-be60-d8c27b330779","Type":"ContainerStarted","Data":"dbc8b63ecca142db717a7446d2914dae72ff9666dd88ed55d86faaa6914a4f9f"}
Oct 30 00:32:14.895704 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:14.895687    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf"
Oct 30 00:32:14.943734 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:14.943678    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf" podStartSLOduration=1.878979296 podStartE2EDuration="4.943662398s" totalImagesPullingTime="3.064683102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:14.942769468 +0000 UTC m=+25.330445858" watchObservedRunningTime="2026-10-30 00:32:14.943662398 +0000 UTC m=+25.331338827"
Oct 30 00:32:16.632074 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:16.632043    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-68948f994f-s2nqw"]
Oct 30 00:32:16.655058 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:16.654644    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68948f994f-s2nqw"
Oct 30 00:32:16.656262 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:16.656215    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68948f994f-s2nqw"]
Oct 30 00:32:16.707579 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:16.707544    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 00:32:16.760631 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:16.760585    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d663ad6d-f013-41b5-ab8d-ac42beb8c693-console-oauth-config\") pod \"console-68948f994f-s2nqw\" (UID: \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\") " pod="openshift-console/console-68948f994f-s2nqw"
Oct 30 00:32:16.760827 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:16.760654    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d663ad6d-f013-41b5-ab8d-ac42beb8c693-oauth-serving-cert\") pod \"console-68948f994f-s2nqw\" (UID: \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\") " pod="openshift-console/console-68948f994f-s2nqw"
Oct 30 00:32:16.760827 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:16.760708    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d663ad6d-f013-41b5-ab8d-ac42beb8c693-console-config\") pod \"console-68948f994f-s2nqw\" (UID: \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\") " pod="openshift-console/console-68948f994f-s2nqw"
Oct 30 00:32:16.761244 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:16.761184    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d663ad6d-f013-41b5-ab8d-ac42beb8c693-service-ca\") pod \"console-68948f994f-s2nqw\" (UID: \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\") " pod="openshift-console/console-68948f994f-s2nqw"
Oct 30 00:32:16.761359 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:16.761292    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d663ad6d-f013-41b5-ab8d-ac42beb8c693-console-serving-cert\") pod \"console-68948f994f-s2nqw\" (UID: \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\") " pod="openshift-console/console-68948f994f-s2nqw"
Oct 30 00:32:16.761359 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:16.761346    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d663ad6d-f013-41b5-ab8d-ac42beb8c693-tmp\") pod \"console-68948f994f-s2nqw\" (UID: \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\") " pod="openshift-console/console-68948f994f-s2nqw"
Oct 30 00:32:16.761475 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:16.761375    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq5c6\" (UniqueName: \"kubernetes.io/projected/d663ad6d-f013-41b5-ab8d-ac42beb8c693-kube-api-access-kq5c6\") pod \"console-68948f994f-s2nqw\" (UID: \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\") " pod="openshift-console/console-68948f994f-s2nqw"
Oct 30 00:32:16.761475 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:16.761465    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d663ad6d-f013-41b5-ab8d-ac42beb8c693-trusted-ca-bundle\") pod \"console-68948f994f-s2nqw\" (UID: \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\") " pod="openshift-console/console-68948f994f-s2nqw"
Oct 30 00:32:16.862347 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:16.862303    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d663ad6d-f013-41b5-ab8d-ac42beb8c693-console-oauth-config\") pod \"console-68948f994f-s2nqw\" (UID: \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\") " pod="openshift-console/console-68948f994f-s2nqw"
Oct 30 00:32:16.862347 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:16.862355    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d663ad6d-f013-41b5-ab8d-ac42beb8c693-oauth-serving-cert\") pod \"console-68948f994f-s2nqw\" (UID: \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\") " pod="openshift-console/console-68948f994f-s2nqw"
Oct 30 00:32:16.862605 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:16.862551    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d663ad6d-f013-41b5-ab8d-ac42beb8c693-console-config\") pod \"console-68948f994f-s2nqw\" (UID: \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\") " pod="openshift-console/console-68948f994f-s2nqw"
Oct 30 00:32:16.862662 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:16.862615    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d663ad6d-f013-41b5-ab8d-ac42beb8c693-service-ca\") pod \"console-68948f994f-s2nqw\" (UID: \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\") " pod="openshift-console/console-68948f994f-s2nqw"
Oct 30 00:32:16.862713 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:16.862672    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d663ad6d-f013-41b5-ab8d-ac42beb8c693-console-serving-cert\") pod \"console-68948f994f-s2nqw\" (UID: \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\") " pod="openshift-console/console-68948f994f-s2nqw"
Oct 30 00:32:16.862713 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:16.862699    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d663ad6d-f013-41b5-ab8d-ac42beb8c693-tmp\") pod \"console-68948f994f-s2nqw\" (UID: \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\") " pod="openshift-console/console-68948f994f-s2nqw"
Oct 30 00:32:16.862846 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:16.862730    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq5c6\" (UniqueName: \"kubernetes.io/projected/d663ad6d-f013-41b5-ab8d-ac42beb8c693-kube-api-access-kq5c6\") pod \"console-68948f994f-s2nqw\" (UID: \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\") " pod="openshift-console/console-68948f994f-s2nqw"
Oct 30 00:32:16.862846 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:16.862765    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d663ad6d-f013-41b5-ab8d-ac42beb8c693-trusted-ca-bundle\") pod \"console-68948f994f-s2nqw\" (UID: \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\") " pod="openshift-console/console-68948f994f-s2nqw"
Oct 30 00:32:16.863180 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:16.863119    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d663ad6d-f013-41b5-ab8d-ac42beb8c693-oauth-serving-cert\") pod \"console-68948f994f-s2nqw\" (UID: \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\") " pod="openshift-console/console-68948f994f-s2nqw"
Oct 30 00:32:16.865412 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:16.863518    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d663ad6d-f013-41b5-ab8d-ac42beb8c693-console-config\") pod \"console-68948f994f-s2nqw\" (UID: \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\") " pod="openshift-console/console-68948f994f-s2nqw"
Oct 30 00:32:16.865412 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:16.864000    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d663ad6d-f013-41b5-ab8d-ac42beb8c693-service-ca\") pod \"console-68948f994f-s2nqw\" (UID: \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\") " pod="openshift-console/console-68948f994f-s2nqw"
Oct 30 00:32:16.865412 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:16.865179    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d663ad6d-f013-41b5-ab8d-ac42beb8c693-trusted-ca-bundle\") pod \"console-68948f994f-s2nqw\" (UID: \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\") " pod="openshift-console/console-68948f994f-s2nqw"
Oct 30 00:32:16.865412 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:16.865304    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d663ad6d-f013-41b5-ab8d-ac42beb8c693-tmp\") pod \"console-68948f994f-s2nqw\" (UID: \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\") " pod="openshift-console/console-68948f994f-s2nqw"
Oct 30 00:32:16.867018 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:16.866994    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d663ad6d-f013-41b5-ab8d-ac42beb8c693-console-serving-cert\") pod \"console-68948f994f-s2nqw\" (UID: \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\") " pod="openshift-console/console-68948f994f-s2nqw"
Oct 30 00:32:16.868093 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:16.868067    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d663ad6d-f013-41b5-ab8d-ac42beb8c693-console-oauth-config\") pod \"console-68948f994f-s2nqw\" (UID: \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\") " pod="openshift-console/console-68948f994f-s2nqw"
Oct 30 00:32:16.895562 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:16.895490    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq5c6\" (UniqueName: \"kubernetes.io/projected/d663ad6d-f013-41b5-ab8d-ac42beb8c693-kube-api-access-kq5c6\") pod \"console-68948f994f-s2nqw\" (UID: \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\") " pod="openshift-console/console-68948f994f-s2nqw"
Oct 30 00:32:16.905903 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:16.905858    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-766c9f88f-jvcvh" event={"ID":"2fa1bcf9-0df3-4978-a8a4-493c6b066681","Type":"ContainerStarted","Data":"f6e005ca0152513957124f940129fd753851241fd1f1f5a255faa55d8c1c78d8"}
Oct 30 00:32:16.980049 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:16.980007    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68948f994f-s2nqw"
Oct 30 00:32:17.303753 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:17.303654    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68948f994f-s2nqw"]
Oct 30 00:32:17.307701 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:32:17.307591    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd663ad6d_f013_41b5_ab8d_ac42beb8c693.slice/crio-65d59ced03f9b9e2818b8e93446d9736dcb95cbcaeb8dbebfefdc8b96a3e7352 WatchSource:0}: Error finding container 65d59ced03f9b9e2818b8e93446d9736dcb95cbcaeb8dbebfefdc8b96a3e7352: Status 404 returned error can't find the container with id 65d59ced03f9b9e2818b8e93446d9736dcb95cbcaeb8dbebfefdc8b96a3e7352
Oct 30 00:32:17.910740 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:17.910703    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68948f994f-s2nqw" event={"ID":"d663ad6d-f013-41b5-ab8d-ac42beb8c693","Type":"ContainerStarted","Data":"d185aef26e1995e1267352b8cf5d939f814793da196ce30d1ab96229ff7c4d99"}
Oct 30 00:32:17.911226 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:17.910752    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68948f994f-s2nqw" event={"ID":"d663ad6d-f013-41b5-ab8d-ac42beb8c693","Type":"ContainerStarted","Data":"65d59ced03f9b9e2818b8e93446d9736dcb95cbcaeb8dbebfefdc8b96a3e7352"}
Oct 30 00:32:17.955134 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:17.955065    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-68948f994f-s2nqw" podStartSLOduration=1.955051749 podStartE2EDuration="1.955051749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:17.954176585 +0000 UTC m=+28.341852974" watchObservedRunningTime="2026-10-30 00:32:17.955051749 +0000 UTC m=+28.342728138"
Oct 30 00:32:18.917870 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:18.917823    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-766c9f88f-jvcvh" event={"ID":"2fa1bcf9-0df3-4978-a8a4-493c6b066681","Type":"ContainerStarted","Data":"71dfae28c277e64aeb0685c553010430c7c3bbb27504a12cc0a7753aa40694b2"}
Oct 30 00:32:18.918382 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:18.917909    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-766c9f88f-jvcvh" event={"ID":"2fa1bcf9-0df3-4978-a8a4-493c6b066681","Type":"ContainerStarted","Data":"cb44a019aeaf0d2d06274e7eb1a1b180180bba5e17adfd2fc0e0638fd02f5026"}
Oct 30 00:32:18.920389 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:18.920358    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-r7bf7" event={"ID":"4a3d031e-b63e-4201-afb9-ec659b67ac1e","Type":"ContainerStarted","Data":"9e5c06e28a1f63dbb7f32ab42c119ed6dba6035b53bb20a14ca6beda5c9b217b"}
Oct 30 00:32:18.920681 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:18.920655    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-r7bf7"
Oct 30 00:32:18.942052 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:18.942018    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-r7bf7"
Oct 30 00:32:19.049638 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:19.049573    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-r7bf7" podStartSLOduration=2.751556531 podStartE2EDuration="20.049555699s" totalImagesPullingTime="17.297999168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:31:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:19.049000915 +0000 UTC m=+29.436677303" watchObservedRunningTime="2026-10-30 00:32:19.049555699 +0000 UTC m=+29.437232085"
Oct 30 00:32:19.050104 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:19.050063    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-766c9f88f-jvcvh" podStartSLOduration=2.360691503 podStartE2EDuration="6.050048312s" totalImagesPullingTime="3.689356809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:19.000597897 +0000 UTC m=+29.388274299" watchObservedRunningTime="2026-10-30 00:32:19.050048312 +0000 UTC m=+29.437724701"
Oct 30 00:32:20.061740 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:20.061702    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68948f994f-s2nqw"]
Oct 30 00:32:21.796968 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:21.796936    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wpvjg"
Oct 30 00:32:21.914013 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:21.913980    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-648d8c566c-nw7rf"
Oct 30 00:32:23.833229 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:23.833193    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7d899fbfd7-2j85r"
Oct 30 00:32:26.520060 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:32:26.520015    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:32:26.980524 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:26.980482    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68948f994f-s2nqw"
Oct 30 00:32:28.551003 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:28.550965    4393 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wzd9_4f7ca654-edeb-4457-9239-8da741b5dafd/konnectivity-agent/0.log" limit=1024
Oct 30 00:32:28.711318 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:28.711283    4393 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6wzd9_4f7ca654-edeb-4457-9239-8da741b5dafd/konnectivity-agent/0.log" limit=1024
Oct 30 00:32:29.268116 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:29.268080    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cc457646f-sjj89"]
Oct 30 00:32:45.086410 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.086349    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-68948f994f-s2nqw" podUID="d663ad6d-f013-41b5-ab8d-ac42beb8c693" containerName="console" containerID="cri-o://d185aef26e1995e1267352b8cf5d939f814793da196ce30d1ab96229ff7c4d99" gracePeriod=15
Oct 30 00:32:45.116727 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:32:45.116690    4393 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd663ad6d_f013_41b5_ab8d_ac42beb8c693.slice/crio-d185aef26e1995e1267352b8cf5d939f814793da196ce30d1ab96229ff7c4d99.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 00:32:45.362617 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.362542    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68948f994f-s2nqw"
Oct 30 00:32:45.515736 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.515697    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d663ad6d-f013-41b5-ab8d-ac42beb8c693-console-config\" (UniqueName: \"kubernetes.io/configmap/d663ad6d-f013-41b5-ab8d-ac42beb8c693-console-config\") pod \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\" (UID: \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\") "
Oct 30 00:32:45.515945 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.515754    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d663ad6d-f013-41b5-ab8d-ac42beb8c693-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d663ad6d-f013-41b5-ab8d-ac42beb8c693-oauth-serving-cert\") pod \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\" (UID: \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\") "
Oct 30 00:32:45.515945 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.515827    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d663ad6d-f013-41b5-ab8d-ac42beb8c693-kube-api-access-kq5c6\" (UniqueName: \"kubernetes.io/projected/d663ad6d-f013-41b5-ab8d-ac42beb8c693-kube-api-access-kq5c6\") pod \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\" (UID: \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\") "
Oct 30 00:32:45.515945 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.515869    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d663ad6d-f013-41b5-ab8d-ac42beb8c693-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d663ad6d-f013-41b5-ab8d-ac42beb8c693-console-oauth-config\") pod \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\" (UID: \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\") "
Oct 30 00:32:45.515945 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.515892    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d663ad6d-f013-41b5-ab8d-ac42beb8c693-service-ca\" (UniqueName: \"kubernetes.io/configmap/d663ad6d-f013-41b5-ab8d-ac42beb8c693-service-ca\") pod \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\" (UID: \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\") "
Oct 30 00:32:45.515945 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.515921    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d663ad6d-f013-41b5-ab8d-ac42beb8c693-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d663ad6d-f013-41b5-ab8d-ac42beb8c693-tmp\") pod \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\" (UID: \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\") "
Oct 30 00:32:45.515945 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.515943    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d663ad6d-f013-41b5-ab8d-ac42beb8c693-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d663ad6d-f013-41b5-ab8d-ac42beb8c693-trusted-ca-bundle\") pod \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\" (UID: \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\") "
Oct 30 00:32:45.516249 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.515976    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d663ad6d-f013-41b5-ab8d-ac42beb8c693-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d663ad6d-f013-41b5-ab8d-ac42beb8c693-console-serving-cert\") pod \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\" (UID: \"d663ad6d-f013-41b5-ab8d-ac42beb8c693\") "
Oct 30 00:32:45.516304 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.516244    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d663ad6d-f013-41b5-ab8d-ac42beb8c693-tmp" pod "d663ad6d-f013-41b5-ab8d-ac42beb8c693" (UID: "d663ad6d-f013-41b5-ab8d-ac42beb8c693"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:32:45.516304 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.516135    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d663ad6d-f013-41b5-ab8d-ac42beb8c693-console-config" pod "d663ad6d-f013-41b5-ab8d-ac42beb8c693" (UID: "d663ad6d-f013-41b5-ab8d-ac42beb8c693"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:32:45.516406 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.516291    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d663ad6d-f013-41b5-ab8d-ac42beb8c693-oauth-serving-cert" pod "d663ad6d-f013-41b5-ab8d-ac42beb8c693" (UID: "d663ad6d-f013-41b5-ab8d-ac42beb8c693"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:32:45.516406 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.516338    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d663ad6d-f013-41b5-ab8d-ac42beb8c693-trusted-ca-bundle" pod "d663ad6d-f013-41b5-ab8d-ac42beb8c693" (UID: "d663ad6d-f013-41b5-ab8d-ac42beb8c693"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:32:45.516406 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.516391    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d663ad6d-f013-41b5-ab8d-ac42beb8c693-service-ca" pod "d663ad6d-f013-41b5-ab8d-ac42beb8c693" (UID: "d663ad6d-f013-41b5-ab8d-ac42beb8c693"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:32:45.518522 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.518492    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d663ad6d-f013-41b5-ab8d-ac42beb8c693-console-serving-cert" pod "d663ad6d-f013-41b5-ab8d-ac42beb8c693" (UID: "d663ad6d-f013-41b5-ab8d-ac42beb8c693"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:32:45.518522 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.518511    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d663ad6d-f013-41b5-ab8d-ac42beb8c693-kube-api-access-kq5c6" pod "d663ad6d-f013-41b5-ab8d-ac42beb8c693" (UID: "d663ad6d-f013-41b5-ab8d-ac42beb8c693"). InnerVolumeSpecName "kube-api-access-kq5c6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:32:45.518522 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.518514    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d663ad6d-f013-41b5-ab8d-ac42beb8c693-console-oauth-config" pod "d663ad6d-f013-41b5-ab8d-ac42beb8c693" (UID: "d663ad6d-f013-41b5-ab8d-ac42beb8c693"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:32:45.561325 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.561289    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-77577bccf6-6vmfm"]
Oct 30 00:32:45.561658 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.561640    4393 cpu_manager.go:414] "containerMap: removing container" podUID="d663ad6d-f013-41b5-ab8d-ac42beb8c693" containerName="console"
Oct 30 00:32:45.561738 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.561660    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d663ad6d-f013-41b5-ab8d-ac42beb8c693" containerName="console"
Oct 30 00:32:45.561809 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.561746    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="d663ad6d-f013-41b5-ab8d-ac42beb8c693" containerName="console"
Oct 30 00:32:45.584721 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.584690    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77577bccf6-6vmfm"]
Oct 30 00:32:45.584901 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.584817    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77577bccf6-6vmfm"
Oct 30 00:32:45.617119 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.617053    4393 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d663ad6d-f013-41b5-ab8d-ac42beb8c693-oauth-serving-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:32:45.617119 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.617078    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq5c6\" (UniqueName: \"kubernetes.io/projected/d663ad6d-f013-41b5-ab8d-ac42beb8c693-kube-api-access-kq5c6\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:32:45.617119 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.617089    4393 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d663ad6d-f013-41b5-ab8d-ac42beb8c693-console-oauth-config\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:32:45.617119 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.617099    4393 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d663ad6d-f013-41b5-ab8d-ac42beb8c693-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:32:45.617119 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.617108    4393 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d663ad6d-f013-41b5-ab8d-ac42beb8c693-tmp\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:32:45.617119 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.617122    4393 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d663ad6d-f013-41b5-ab8d-ac42beb8c693-trusted-ca-bundle\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:32:45.617389 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.617132    4393 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d663ad6d-f013-41b5-ab8d-ac42beb8c693-console-serving-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:32:45.617389 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.617140    4393 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d663ad6d-f013-41b5-ab8d-ac42beb8c693-console-config\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:32:45.721568 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.718238    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5aedfc1-943a-43a7-bd61-a58892dd8428-tmp\") pod \"console-77577bccf6-6vmfm\" (UID: \"e5aedfc1-943a-43a7-bd61-a58892dd8428\") " pod="openshift-console/console-77577bccf6-6vmfm"
Oct 30 00:32:45.721568 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.718313    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc8ch\" (UniqueName: \"kubernetes.io/projected/e5aedfc1-943a-43a7-bd61-a58892dd8428-kube-api-access-nc8ch\") pod \"console-77577bccf6-6vmfm\" (UID: \"e5aedfc1-943a-43a7-bd61-a58892dd8428\") " pod="openshift-console/console-77577bccf6-6vmfm"
Oct 30 00:32:45.721568 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.718358    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e5aedfc1-943a-43a7-bd61-a58892dd8428-oauth-serving-cert\") pod \"console-77577bccf6-6vmfm\" (UID: \"e5aedfc1-943a-43a7-bd61-a58892dd8428\") " pod="openshift-console/console-77577bccf6-6vmfm"
Oct 30 00:32:45.721568 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.718399    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e5aedfc1-943a-43a7-bd61-a58892dd8428-service-ca\") pod \"console-77577bccf6-6vmfm\" (UID: \"e5aedfc1-943a-43a7-bd61-a58892dd8428\") " pod="openshift-console/console-77577bccf6-6vmfm"
Oct 30 00:32:45.721568 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.718450    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e5aedfc1-943a-43a7-bd61-a58892dd8428-console-oauth-config\") pod \"console-77577bccf6-6vmfm\" (UID: \"e5aedfc1-943a-43a7-bd61-a58892dd8428\") " pod="openshift-console/console-77577bccf6-6vmfm"
Oct 30 00:32:45.721568 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.718509    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e5aedfc1-943a-43a7-bd61-a58892dd8428-console-serving-cert\") pod \"console-77577bccf6-6vmfm\" (UID: \"e5aedfc1-943a-43a7-bd61-a58892dd8428\") " pod="openshift-console/console-77577bccf6-6vmfm"
Oct 30 00:32:45.721568 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.718541    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5aedfc1-943a-43a7-bd61-a58892dd8428-trusted-ca-bundle\") pod \"console-77577bccf6-6vmfm\" (UID: \"e5aedfc1-943a-43a7-bd61-a58892dd8428\") " pod="openshift-console/console-77577bccf6-6vmfm"
Oct 30 00:32:45.721568 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.718592    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e5aedfc1-943a-43a7-bd61-a58892dd8428-console-config\") pod \"console-77577bccf6-6vmfm\" (UID: \"e5aedfc1-943a-43a7-bd61-a58892dd8428\") " pod="openshift-console/console-77577bccf6-6vmfm"
Oct 30 00:32:45.819929 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.819886    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e5aedfc1-943a-43a7-bd61-a58892dd8428-oauth-serving-cert\") pod \"console-77577bccf6-6vmfm\" (UID: \"e5aedfc1-943a-43a7-bd61-a58892dd8428\") " pod="openshift-console/console-77577bccf6-6vmfm"
Oct 30 00:32:45.820125 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.819935    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e5aedfc1-943a-43a7-bd61-a58892dd8428-service-ca\") pod \"console-77577bccf6-6vmfm\" (UID: \"e5aedfc1-943a-43a7-bd61-a58892dd8428\") " pod="openshift-console/console-77577bccf6-6vmfm"
Oct 30 00:32:45.820125 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.819978    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e5aedfc1-943a-43a7-bd61-a58892dd8428-console-oauth-config\") pod \"console-77577bccf6-6vmfm\" (UID: \"e5aedfc1-943a-43a7-bd61-a58892dd8428\") " pod="openshift-console/console-77577bccf6-6vmfm"
Oct 30 00:32:45.820125 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.820022    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e5aedfc1-943a-43a7-bd61-a58892dd8428-console-serving-cert\") pod \"console-77577bccf6-6vmfm\" (UID: \"e5aedfc1-943a-43a7-bd61-a58892dd8428\") " pod="openshift-console/console-77577bccf6-6vmfm"
Oct 30 00:32:45.820125 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.820042    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5aedfc1-943a-43a7-bd61-a58892dd8428-trusted-ca-bundle\") pod \"console-77577bccf6-6vmfm\" (UID: \"e5aedfc1-943a-43a7-bd61-a58892dd8428\") " pod="openshift-console/console-77577bccf6-6vmfm"
Oct 30 00:32:45.820125 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.820073    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e5aedfc1-943a-43a7-bd61-a58892dd8428-console-config\") pod \"console-77577bccf6-6vmfm\" (UID: \"e5aedfc1-943a-43a7-bd61-a58892dd8428\") " pod="openshift-console/console-77577bccf6-6vmfm"
Oct 30 00:32:45.820125 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.820102    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5aedfc1-943a-43a7-bd61-a58892dd8428-tmp\") pod \"console-77577bccf6-6vmfm\" (UID: \"e5aedfc1-943a-43a7-bd61-a58892dd8428\") " pod="openshift-console/console-77577bccf6-6vmfm"
Oct 30 00:32:45.820433 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.820138    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc8ch\" (UniqueName: \"kubernetes.io/projected/e5aedfc1-943a-43a7-bd61-a58892dd8428-kube-api-access-nc8ch\") pod \"console-77577bccf6-6vmfm\" (UID: \"e5aedfc1-943a-43a7-bd61-a58892dd8428\") " pod="openshift-console/console-77577bccf6-6vmfm"
Oct 30 00:32:45.820764 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.820698    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e5aedfc1-943a-43a7-bd61-a58892dd8428-oauth-serving-cert\") pod \"console-77577bccf6-6vmfm\" (UID: \"e5aedfc1-943a-43a7-bd61-a58892dd8428\") " pod="openshift-console/console-77577bccf6-6vmfm"
Oct 30 00:32:45.821062 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.820760    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e5aedfc1-943a-43a7-bd61-a58892dd8428-tmp\") pod \"console-77577bccf6-6vmfm\" (UID: \"e5aedfc1-943a-43a7-bd61-a58892dd8428\") " pod="openshift-console/console-77577bccf6-6vmfm"
Oct 30 00:32:45.821156 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.821132    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e5aedfc1-943a-43a7-bd61-a58892dd8428-service-ca\") pod \"console-77577bccf6-6vmfm\" (UID: \"e5aedfc1-943a-43a7-bd61-a58892dd8428\") " pod="openshift-console/console-77577bccf6-6vmfm"
Oct 30 00:32:45.821391 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.821365    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e5aedfc1-943a-43a7-bd61-a58892dd8428-console-config\") pod \"console-77577bccf6-6vmfm\" (UID: \"e5aedfc1-943a-43a7-bd61-a58892dd8428\") " pod="openshift-console/console-77577bccf6-6vmfm"
Oct 30 00:32:45.821484 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.821431    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5aedfc1-943a-43a7-bd61-a58892dd8428-trusted-ca-bundle\") pod \"console-77577bccf6-6vmfm\" (UID: \"e5aedfc1-943a-43a7-bd61-a58892dd8428\") " pod="openshift-console/console-77577bccf6-6vmfm"
Oct 30 00:32:45.822731 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.822709    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e5aedfc1-943a-43a7-bd61-a58892dd8428-console-serving-cert\") pod \"console-77577bccf6-6vmfm\" (UID: \"e5aedfc1-943a-43a7-bd61-a58892dd8428\") " pod="openshift-console/console-77577bccf6-6vmfm"
Oct 30 00:32:45.822825 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.822741    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e5aedfc1-943a-43a7-bd61-a58892dd8428-console-oauth-config\") pod \"console-77577bccf6-6vmfm\" (UID: \"e5aedfc1-943a-43a7-bd61-a58892dd8428\") " pod="openshift-console/console-77577bccf6-6vmfm"
Oct 30 00:32:45.843261 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.843239    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc8ch\" (UniqueName: \"kubernetes.io/projected/e5aedfc1-943a-43a7-bd61-a58892dd8428-kube-api-access-nc8ch\") pod \"console-77577bccf6-6vmfm\" (UID: \"e5aedfc1-943a-43a7-bd61-a58892dd8428\") " pod="openshift-console/console-77577bccf6-6vmfm"
Oct 30 00:32:45.907950 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:45.907912    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77577bccf6-6vmfm"
Oct 30 00:32:46.020348 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:46.020319    4393 generic.go:356] "Generic (PLEG): container finished" podID="d663ad6d-f013-41b5-ab8d-ac42beb8c693" containerID="d185aef26e1995e1267352b8cf5d939f814793da196ce30d1ab96229ff7c4d99" exitCode=0
Oct 30 00:32:46.020493 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:46.020378    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68948f994f-s2nqw" event={"ID":"d663ad6d-f013-41b5-ab8d-ac42beb8c693","Type":"ContainerDied","Data":"d185aef26e1995e1267352b8cf5d939f814793da196ce30d1ab96229ff7c4d99"}
Oct 30 00:32:46.020493 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:46.020389    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68948f994f-s2nqw"
Oct 30 00:32:46.020493 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:46.020417    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68948f994f-s2nqw" event={"ID":"d663ad6d-f013-41b5-ab8d-ac42beb8c693","Type":"ContainerDied","Data":"65d59ced03f9b9e2818b8e93446d9736dcb95cbcaeb8dbebfefdc8b96a3e7352"}
Oct 30 00:32:46.020493 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:46.020436    4393 scope.go:122] "RemoveContainer" containerID="d185aef26e1995e1267352b8cf5d939f814793da196ce30d1ab96229ff7c4d99"
Oct 30 00:32:46.028552 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:46.028534    4393 scope.go:122] "RemoveContainer" containerID="d185aef26e1995e1267352b8cf5d939f814793da196ce30d1ab96229ff7c4d99"
Oct 30 00:32:46.028832 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:32:46.028810    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d185aef26e1995e1267352b8cf5d939f814793da196ce30d1ab96229ff7c4d99\": container with ID starting with d185aef26e1995e1267352b8cf5d939f814793da196ce30d1ab96229ff7c4d99 not found: ID does not exist" containerID="d185aef26e1995e1267352b8cf5d939f814793da196ce30d1ab96229ff7c4d99"
Oct 30 00:32:46.028892 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:46.028841    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d185aef26e1995e1267352b8cf5d939f814793da196ce30d1ab96229ff7c4d99"} err="failed to get container status \"d185aef26e1995e1267352b8cf5d939f814793da196ce30d1ab96229ff7c4d99\": rpc error: code = NotFound desc = could not find container \"d185aef26e1995e1267352b8cf5d939f814793da196ce30d1ab96229ff7c4d99\": container with ID starting with d185aef26e1995e1267352b8cf5d939f814793da196ce30d1ab96229ff7c4d99 not found: ID does not exist"
Oct 30 00:32:46.071273 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:46.071243    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68948f994f-s2nqw"]
Oct 30 00:32:46.077062 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:46.077041    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68948f994f-s2nqw"]
Oct 30 00:32:46.230932 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:46.230769    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77577bccf6-6vmfm"]
Oct 30 00:32:46.233334 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:32:46.233307    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5aedfc1_943a_43a7_bd61_a58892dd8428.slice/crio-1271dfff63f521a96a8c9d0ea19fa512d053a95ce99c442c13248e1b3f1cd2af WatchSource:0}: Error finding container 1271dfff63f521a96a8c9d0ea19fa512d053a95ce99c442c13248e1b3f1cd2af: Status 404 returned error can't find the container with id 1271dfff63f521a96a8c9d0ea19fa512d053a95ce99c442c13248e1b3f1cd2af
Oct 30 00:32:47.025827 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:47.025796    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77577bccf6-6vmfm" event={"ID":"e5aedfc1-943a-43a7-bd61-a58892dd8428","Type":"ContainerStarted","Data":"9bbe594880510407b6c9941abd0fb21dcfebb0e2a294fe3a9165f12087a4d87f"}
Oct 30 00:32:47.025992 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:47.025839    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77577bccf6-6vmfm" event={"ID":"e5aedfc1-943a-43a7-bd61-a58892dd8428","Type":"ContainerStarted","Data":"1271dfff63f521a96a8c9d0ea19fa512d053a95ce99c442c13248e1b3f1cd2af"}
Oct 30 00:32:47.087823 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:47.087743    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-77577bccf6-6vmfm" podStartSLOduration=2.087730355 podStartE2EDuration="2.087730355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:47.07541495 +0000 UTC m=+57.463091338" watchObservedRunningTime="2026-10-30 00:32:47.087730355 +0000 UTC m=+57.475406738"
Oct 30 00:32:47.721909 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:47.721877    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d663ad6d-f013-41b5-ab8d-ac42beb8c693" path="/var/lib/kubelet/pods/d663ad6d-f013-41b5-ab8d-ac42beb8c693/volumes"
Oct 30 00:32:49.141384 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.141318    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 00:32:49.172722 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.172688    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 00:32:49.172915 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.172838    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.219237 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.219194    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 00:32:49.219237 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.219220    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 00:32:49.219237 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.219236    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 00:32:49.219492 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.219194    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 00:32:49.219492 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.219262    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 00:32:49.219492 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.219194    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 00:32:49.219492 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.219228    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 00:32:49.219492 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.219194    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 00:32:49.219492 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.219196    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 00:32:49.219492 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.219347    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-5vxhs\""
Oct 30 00:32:49.247565 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.247536    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/89bbffef-b6f8-44df-9752-b1c163a9abea-config-volume\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.247565 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.247570    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89bbffef-b6f8-44df-9752-b1c163a9abea-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.247768 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.247587    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/89bbffef-b6f8-44df-9752-b1c163a9abea-tls-assets\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.247768 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.247605    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/89bbffef-b6f8-44df-9752-b1c163a9abea-web-config\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.247768 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.247661    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/89bbffef-b6f8-44df-9752-b1c163a9abea-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.247768 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.247695    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89bbffef-b6f8-44df-9752-b1c163a9abea-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.247768 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.247748    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/89bbffef-b6f8-44df-9752-b1c163a9abea-config-out\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.247960 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.247823    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cfh2\" (UniqueName: \"kubernetes.io/projected/89bbffef-b6f8-44df-9752-b1c163a9abea-kube-api-access-4cfh2\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.247960 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.247864    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/89bbffef-b6f8-44df-9752-b1c163a9abea-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.247960 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.247881    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/89bbffef-b6f8-44df-9752-b1c163a9abea-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.247960 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.247926    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/89bbffef-b6f8-44df-9752-b1c163a9abea-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.247960 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.247955    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/89bbffef-b6f8-44df-9752-b1c163a9abea-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.248104 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.247975    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/89bbffef-b6f8-44df-9752-b1c163a9abea-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.348880 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.348834    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/89bbffef-b6f8-44df-9752-b1c163a9abea-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.348880 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.348881    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/89bbffef-b6f8-44df-9752-b1c163a9abea-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.349104 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.348911    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/89bbffef-b6f8-44df-9752-b1c163a9abea-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.349141 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.349103    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/89bbffef-b6f8-44df-9752-b1c163a9abea-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.349192 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.349146    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/89bbffef-b6f8-44df-9752-b1c163a9abea-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.349245 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.349191    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/89bbffef-b6f8-44df-9752-b1c163a9abea-config-volume\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.349245 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.349210    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89bbffef-b6f8-44df-9752-b1c163a9abea-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.349245 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.349234    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/89bbffef-b6f8-44df-9752-b1c163a9abea-tls-assets\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.349396 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.349263    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/89bbffef-b6f8-44df-9752-b1c163a9abea-web-config\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.349396 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.349299    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/89bbffef-b6f8-44df-9752-b1c163a9abea-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.349396 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.349331    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89bbffef-b6f8-44df-9752-b1c163a9abea-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.349396 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.349356    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/89bbffef-b6f8-44df-9752-b1c163a9abea-config-out\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.349396 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.349389    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cfh2\" (UniqueName: \"kubernetes.io/projected/89bbffef-b6f8-44df-9752-b1c163a9abea-kube-api-access-4cfh2\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.349645 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.349519    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/89bbffef-b6f8-44df-9752-b1c163a9abea-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.350668 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.350640    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89bbffef-b6f8-44df-9752-b1c163a9abea-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.351239 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.351214    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89bbffef-b6f8-44df-9752-b1c163a9abea-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.352420 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.352393    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/89bbffef-b6f8-44df-9752-b1c163a9abea-config-volume\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.353066 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.352648    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/89bbffef-b6f8-44df-9752-b1c163a9abea-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.353066 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.352849    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/89bbffef-b6f8-44df-9752-b1c163a9abea-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.353066 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.353003    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/89bbffef-b6f8-44df-9752-b1c163a9abea-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.353238 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.353082    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/89bbffef-b6f8-44df-9752-b1c163a9abea-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.353238 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.353146    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/89bbffef-b6f8-44df-9752-b1c163a9abea-tls-assets\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.353529 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.353500    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/89bbffef-b6f8-44df-9752-b1c163a9abea-web-config\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.353797 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.353755    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/89bbffef-b6f8-44df-9752-b1c163a9abea-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.354269 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.354254    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/89bbffef-b6f8-44df-9752-b1c163a9abea-config-out\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.372502 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.372468    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cfh2\" (UniqueName: \"kubernetes.io/projected/89bbffef-b6f8-44df-9752-b1c163a9abea-kube-api-access-4cfh2\") pod \"alertmanager-main-0\" (UID: \"89bbffef-b6f8-44df-9752-b1c163a9abea\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.583452 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.583357    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 00:32:49.946075 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:49.946041    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 00:32:49.949584 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:32:49.949554    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89bbffef_b6f8_44df_9752_b1c163a9abea.slice/crio-4374336bc71a4324e9d1cf7019e208cbfcb1044d883f6bba0144e204c3cf6935 WatchSource:0}: Error finding container 4374336bc71a4324e9d1cf7019e208cbfcb1044d883f6bba0144e204c3cf6935: Status 404 returned error can't find the container with id 4374336bc71a4324e9d1cf7019e208cbfcb1044d883f6bba0144e204c3cf6935
Oct 30 00:32:50.040459 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:50.040424    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89bbffef-b6f8-44df-9752-b1c163a9abea","Type":"ContainerStarted","Data":"4374336bc71a4324e9d1cf7019e208cbfcb1044d883f6bba0144e204c3cf6935"}
Oct 30 00:32:51.049800 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:51.049743    4393 generic.go:356] "Generic (PLEG): container finished" podID="89bbffef-b6f8-44df-9752-b1c163a9abea" containerID="276a4ac58521647b600467a745c2dae0010b5a265d9e5c4e4c87cc3f97c9d6cd" exitCode=0
Oct 30 00:32:51.050200 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:51.049831    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89bbffef-b6f8-44df-9752-b1c163a9abea","Type":"ContainerDied","Data":"276a4ac58521647b600467a745c2dae0010b5a265d9e5c4e4c87cc3f97c9d6cd"}
Oct 30 00:32:54.064329 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:54.064294    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89bbffef-b6f8-44df-9752-b1c163a9abea","Type":"ContainerStarted","Data":"f848e23a990dd03a812b54b9c6720d7bcfb1349dc678d7d4b460eabf5ef87dc3"}
Oct 30 00:32:54.064714 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:54.064358    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89bbffef-b6f8-44df-9752-b1c163a9abea","Type":"ContainerStarted","Data":"55d056cb3306d4a33a37eaab407dc3192e3e64e34b05d91377166427b89fa9a0"}
Oct 30 00:32:54.064714 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:54.064386    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89bbffef-b6f8-44df-9752-b1c163a9abea","Type":"ContainerStarted","Data":"8544f8562ea93f7709c695656602059adb968652e6a409ea8d7859f4dd62ca36"}
Oct 30 00:32:54.064714 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:54.064409    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89bbffef-b6f8-44df-9752-b1c163a9abea","Type":"ContainerStarted","Data":"943b9bc40f18d3353a9daf5e5d9c80c7424bf7c5afedb07de840b4d291d36433"}
Oct 30 00:32:54.064714 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:54.064438    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89bbffef-b6f8-44df-9752-b1c163a9abea","Type":"ContainerStarted","Data":"c79bc8716ae3202fbb83dc81a79cfd6c13d7c9c8b12e2fe6daeda1ad2ab8345f"}
Oct 30 00:32:54.064714 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:54.064461    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"89bbffef-b6f8-44df-9752-b1c163a9abea","Type":"ContainerStarted","Data":"1d96b07d20cdd212db59e585ad8ed33dfaa3676d4dc4cd5ccbb669f2112e1a6f"}
Oct 30 00:32:54.145412 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:54.145330    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.208154054 podStartE2EDuration="5.145308803s" totalImagesPullingTime="1.922417067s" totalInitContainerRuntime="14.737682ms" isStatefulPod=false podCreationTimestamp="2026-10-30 00:32:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:32:54.132632585 +0000 UTC m=+64.520308987" watchObservedRunningTime="2026-10-30 00:32:54.145308803 +0000 UTC m=+64.532985194"
Oct 30 00:32:54.288512 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:54.288453    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7cc457646f-sjj89" podUID="72a65fa9-4477-434a-aaac-0394505927e5" containerName="console" containerID="cri-o://de2caceb2c2f9c010a09716d4a14759468bde844cff71f7d34f33b6bae84745d" gracePeriod=15
Oct 30 00:32:54.578594 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:54.578509    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cc457646f-sjj89"
Oct 30 00:32:54.697652 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:54.697614    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72a65fa9-4477-434a-aaac-0394505927e5-tmp\" (UniqueName: \"kubernetes.io/empty-dir/72a65fa9-4477-434a-aaac-0394505927e5-tmp\") pod \"72a65fa9-4477-434a-aaac-0394505927e5\" (UID: \"72a65fa9-4477-434a-aaac-0394505927e5\") "
Oct 30 00:32:54.697907 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:54.697718    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72a65fa9-4477-434a-aaac-0394505927e5-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72a65fa9-4477-434a-aaac-0394505927e5-console-serving-cert\") pod \"72a65fa9-4477-434a-aaac-0394505927e5\" (UID: \"72a65fa9-4477-434a-aaac-0394505927e5\") "
Oct 30 00:32:54.697907 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:54.697754    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72a65fa9-4477-434a-aaac-0394505927e5-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72a65fa9-4477-434a-aaac-0394505927e5-oauth-serving-cert\") pod \"72a65fa9-4477-434a-aaac-0394505927e5\" (UID: \"72a65fa9-4477-434a-aaac-0394505927e5\") "
Oct 30 00:32:54.697907 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:54.697808    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72a65fa9-4477-434a-aaac-0394505927e5-console-config\" (UniqueName: \"kubernetes.io/configmap/72a65fa9-4477-434a-aaac-0394505927e5-console-config\") pod \"72a65fa9-4477-434a-aaac-0394505927e5\" (UID: \"72a65fa9-4477-434a-aaac-0394505927e5\") "
Oct 30 00:32:54.697907 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:54.697862    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72a65fa9-4477-434a-aaac-0394505927e5-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72a65fa9-4477-434a-aaac-0394505927e5-console-oauth-config\") pod \"72a65fa9-4477-434a-aaac-0394505927e5\" (UID: \"72a65fa9-4477-434a-aaac-0394505927e5\") "
Oct 30 00:32:54.697907 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:54.697888    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72a65fa9-4477-434a-aaac-0394505927e5-kube-api-access-l9cbk\" (UniqueName: \"kubernetes.io/projected/72a65fa9-4477-434a-aaac-0394505927e5-kube-api-access-l9cbk\") pod \"72a65fa9-4477-434a-aaac-0394505927e5\" (UID: \"72a65fa9-4477-434a-aaac-0394505927e5\") "
Oct 30 00:32:54.698155 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:54.697917    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72a65fa9-4477-434a-aaac-0394505927e5-service-ca\" (UniqueName: \"kubernetes.io/configmap/72a65fa9-4477-434a-aaac-0394505927e5-service-ca\") pod \"72a65fa9-4477-434a-aaac-0394505927e5\" (UID: \"72a65fa9-4477-434a-aaac-0394505927e5\") "
Oct 30 00:32:54.698155 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:54.698035    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a65fa9-4477-434a-aaac-0394505927e5-tmp" pod "72a65fa9-4477-434a-aaac-0394505927e5" (UID: "72a65fa9-4477-434a-aaac-0394505927e5"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:32:54.698434 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:54.698281    4393 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72a65fa9-4477-434a-aaac-0394505927e5-tmp\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:32:54.698532 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:54.698441    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72a65fa9-4477-434a-aaac-0394505927e5-console-config" pod "72a65fa9-4477-434a-aaac-0394505927e5" (UID: "72a65fa9-4477-434a-aaac-0394505927e5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:32:54.698532 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:54.698371    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72a65fa9-4477-434a-aaac-0394505927e5-oauth-serving-cert" pod "72a65fa9-4477-434a-aaac-0394505927e5" (UID: "72a65fa9-4477-434a-aaac-0394505927e5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:32:54.698713 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:54.698693    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72a65fa9-4477-434a-aaac-0394505927e5-service-ca" pod "72a65fa9-4477-434a-aaac-0394505927e5" (UID: "72a65fa9-4477-434a-aaac-0394505927e5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:32:54.700216 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:54.700188    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72a65fa9-4477-434a-aaac-0394505927e5-console-serving-cert" pod "72a65fa9-4477-434a-aaac-0394505927e5" (UID: "72a65fa9-4477-434a-aaac-0394505927e5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:32:54.700330 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:54.700298    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72a65fa9-4477-434a-aaac-0394505927e5-console-oauth-config" pod "72a65fa9-4477-434a-aaac-0394505927e5" (UID: "72a65fa9-4477-434a-aaac-0394505927e5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:32:54.700373 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:54.700344    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a65fa9-4477-434a-aaac-0394505927e5-kube-api-access-l9cbk" pod "72a65fa9-4477-434a-aaac-0394505927e5" (UID: "72a65fa9-4477-434a-aaac-0394505927e5"). InnerVolumeSpecName "kube-api-access-l9cbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:32:54.799055 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:54.799000    4393 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72a65fa9-4477-434a-aaac-0394505927e5-console-serving-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:32:54.799055 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:54.799048    4393 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72a65fa9-4477-434a-aaac-0394505927e5-oauth-serving-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:32:54.799055 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:54.799059    4393 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72a65fa9-4477-434a-aaac-0394505927e5-console-config\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:32:54.799055 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:54.799069    4393 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72a65fa9-4477-434a-aaac-0394505927e5-console-oauth-config\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:32:54.799055 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:54.799078    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9cbk\" (UniqueName: \"kubernetes.io/projected/72a65fa9-4477-434a-aaac-0394505927e5-kube-api-access-l9cbk\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:32:54.799433 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:54.799087    4393 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72a65fa9-4477-434a-aaac-0394505927e5-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:32:55.069246 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:55.069209    4393 generic.go:356] "Generic (PLEG): container finished" podID="72a65fa9-4477-434a-aaac-0394505927e5" containerID="de2caceb2c2f9c010a09716d4a14759468bde844cff71f7d34f33b6bae84745d" exitCode=0
Oct 30 00:32:55.069762 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:55.069259    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cc457646f-sjj89"
Oct 30 00:32:55.069762 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:55.069295    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cc457646f-sjj89" event={"ID":"72a65fa9-4477-434a-aaac-0394505927e5","Type":"ContainerDied","Data":"de2caceb2c2f9c010a09716d4a14759468bde844cff71f7d34f33b6bae84745d"}
Oct 30 00:32:55.069762 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:55.069352    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cc457646f-sjj89" event={"ID":"72a65fa9-4477-434a-aaac-0394505927e5","Type":"ContainerDied","Data":"be1211d9c46ab1f26d76df43bb0e4726e0a8f5eecc90f2bb8208974a0a0da7fe"}
Oct 30 00:32:55.069762 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:55.069380    4393 scope.go:122] "RemoveContainer" containerID="de2caceb2c2f9c010a09716d4a14759468bde844cff71f7d34f33b6bae84745d"
Oct 30 00:32:55.078492 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:55.078470    4393 scope.go:122] "RemoveContainer" containerID="de2caceb2c2f9c010a09716d4a14759468bde844cff71f7d34f33b6bae84745d"
Oct 30 00:32:55.078820 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:32:55.078796    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de2caceb2c2f9c010a09716d4a14759468bde844cff71f7d34f33b6bae84745d\": container with ID starting with de2caceb2c2f9c010a09716d4a14759468bde844cff71f7d34f33b6bae84745d not found: ID does not exist" containerID="de2caceb2c2f9c010a09716d4a14759468bde844cff71f7d34f33b6bae84745d"
Oct 30 00:32:55.078916 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:55.078838    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de2caceb2c2f9c010a09716d4a14759468bde844cff71f7d34f33b6bae84745d"} err="failed to get container status \"de2caceb2c2f9c010a09716d4a14759468bde844cff71f7d34f33b6bae84745d\": rpc error: code = NotFound desc = could not find container \"de2caceb2c2f9c010a09716d4a14759468bde844cff71f7d34f33b6bae84745d\": container with ID starting with de2caceb2c2f9c010a09716d4a14759468bde844cff71f7d34f33b6bae84745d not found: ID does not exist"
Oct 30 00:32:55.118237 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:55.118201    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cc457646f-sjj89"]
Oct 30 00:32:55.130096 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:55.130063    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7cc457646f-sjj89"]
Oct 30 00:32:55.722065 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:55.722032    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72a65fa9-4477-434a-aaac-0394505927e5" path="/var/lib/kubelet/pods/72a65fa9-4477-434a-aaac-0394505927e5/volumes"
Oct 30 00:32:55.908979 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:55.908939    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77577bccf6-6vmfm"
Oct 30 00:32:55.909173 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:55.909029    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77577bccf6-6vmfm"
Oct 30 00:32:55.913961 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:55.913941    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77577bccf6-6vmfm"
Oct 30 00:32:56.079751 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:32:56.079672    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77577bccf6-6vmfm"
Oct 30 00:32:56.510415 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:32:56.510380    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:33:26.512908 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:33:26.512868    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:33:56.508913 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:33:56.508876    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:34:26.509372 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:34:26.509329    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:34:46.660836 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:34:46.660798    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgdm2f"]
Oct 30 00:34:46.661268 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:34:46.661092    4393 cpu_manager.go:414] "containerMap: removing container" podUID="72a65fa9-4477-434a-aaac-0394505927e5" containerName="console"
Oct 30 00:34:46.661268 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:34:46.661101    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72a65fa9-4477-434a-aaac-0394505927e5" containerName="console"
Oct 30 00:34:46.661268 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:34:46.661165    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="72a65fa9-4477-434a-aaac-0394505927e5" containerName="console"
Oct 30 00:34:46.663041 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:34:46.663016    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgdm2f"
Oct 30 00:34:46.689334 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:34:46.689302    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-99ddb\""
Oct 30 00:34:46.689334 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:34:46.689302    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 00:34:46.689546 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:34:46.689300    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 00:34:46.691221 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:34:46.691199    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgdm2f"]
Oct 30 00:34:46.792123 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:34:46.792085    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92nv9\" (UniqueName: \"kubernetes.io/projected/b4dcc75d-f832-4d5d-86ae-baade25bacd1-kube-api-access-92nv9\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgdm2f\" (UID: \"b4dcc75d-f832-4d5d-86ae-baade25bacd1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgdm2f"
Oct 30 00:34:46.792332 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:34:46.792160    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4dcc75d-f832-4d5d-86ae-baade25bacd1-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgdm2f\" (UID: \"b4dcc75d-f832-4d5d-86ae-baade25bacd1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgdm2f"
Oct 30 00:34:46.792332 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:34:46.792270    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4dcc75d-f832-4d5d-86ae-baade25bacd1-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgdm2f\" (UID: \"b4dcc75d-f832-4d5d-86ae-baade25bacd1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgdm2f"
Oct 30 00:34:46.893700 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:34:46.893664    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4dcc75d-f832-4d5d-86ae-baade25bacd1-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgdm2f\" (UID: \"b4dcc75d-f832-4d5d-86ae-baade25bacd1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgdm2f"
Oct 30 00:34:46.893886 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:34:46.893711    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4dcc75d-f832-4d5d-86ae-baade25bacd1-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgdm2f\" (UID: \"b4dcc75d-f832-4d5d-86ae-baade25bacd1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgdm2f"
Oct 30 00:34:46.893886 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:34:46.893738    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92nv9\" (UniqueName: \"kubernetes.io/projected/b4dcc75d-f832-4d5d-86ae-baade25bacd1-kube-api-access-92nv9\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgdm2f\" (UID: \"b4dcc75d-f832-4d5d-86ae-baade25bacd1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgdm2f"
Oct 30 00:34:46.894150 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:34:46.894127    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4dcc75d-f832-4d5d-86ae-baade25bacd1-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgdm2f\" (UID: \"b4dcc75d-f832-4d5d-86ae-baade25bacd1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgdm2f"
Oct 30 00:34:46.894216 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:34:46.894139    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4dcc75d-f832-4d5d-86ae-baade25bacd1-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgdm2f\" (UID: \"b4dcc75d-f832-4d5d-86ae-baade25bacd1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgdm2f"
Oct 30 00:34:46.917443 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:34:46.917373    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92nv9\" (UniqueName: \"kubernetes.io/projected/b4dcc75d-f832-4d5d-86ae-baade25bacd1-kube-api-access-92nv9\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgdm2f\" (UID: \"b4dcc75d-f832-4d5d-86ae-baade25bacd1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgdm2f"
Oct 30 00:34:46.975361 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:34:46.975321    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgdm2f"
Oct 30 00:34:47.271389 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:34:47.271309    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgdm2f"]
Oct 30 00:34:47.274215 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:34:47.274186    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4dcc75d_f832_4d5d_86ae_baade25bacd1.slice/crio-9071da14889eadee34f212e762204e1815d3780277c3b92bb9846499c9939a9a WatchSource:0}: Error finding container 9071da14889eadee34f212e762204e1815d3780277c3b92bb9846499c9939a9a: Status 404 returned error can't find the container with id 9071da14889eadee34f212e762204e1815d3780277c3b92bb9846499c9939a9a
Oct 30 00:34:47.485741 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:34:47.485706    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgdm2f" event={"ID":"b4dcc75d-f832-4d5d-86ae-baade25bacd1","Type":"ContainerStarted","Data":"9071da14889eadee34f212e762204e1815d3780277c3b92bb9846499c9939a9a"}
Oct 30 00:34:53.511465 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:34:53.511430    4393 generic.go:356] "Generic (PLEG): container finished" podID="b4dcc75d-f832-4d5d-86ae-baade25bacd1" containerID="c2991b4afb89c21dfb59c5c8c8e14e0ac987aa162137af68f09d482f25413c55" exitCode=0
Oct 30 00:34:53.511908 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:34:53.511515    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgdm2f" event={"ID":"b4dcc75d-f832-4d5d-86ae-baade25bacd1","Type":"ContainerDied","Data":"c2991b4afb89c21dfb59c5c8c8e14e0ac987aa162137af68f09d482f25413c55"}
Oct 30 00:34:56.508267 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:34:56.508223    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:34:56.524130 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:34:56.524093    4393 generic.go:356] "Generic (PLEG): container finished" podID="b4dcc75d-f832-4d5d-86ae-baade25bacd1" containerID="e5793ae5a44a4abe2b5a53b8f17cc47ba4302137cfea216fc0f6691137459521" exitCode=0
Oct 30 00:34:56.524278 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:34:56.524169    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgdm2f" event={"ID":"b4dcc75d-f832-4d5d-86ae-baade25bacd1","Type":"ContainerDied","Data":"e5793ae5a44a4abe2b5a53b8f17cc47ba4302137cfea216fc0f6691137459521"}
Oct 30 00:35:02.551197 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:02.551162    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgdm2f" event={"ID":"b4dcc75d-f832-4d5d-86ae-baade25bacd1","Type":"ContainerStarted","Data":"90b684f6785871f295969d983cd61340cf9ed4d014d336261f295f0d631aef93"}
Oct 30 00:35:02.593740 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:02.593692    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgdm2f" podStartSLOduration=3.255188415 podStartE2EDuration="16.593678051s" totalImagesPullingTime="13.338489636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:34:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:35:02.593649245 +0000 UTC m=+192.981325633" watchObservedRunningTime="2026-10-30 00:35:02.593678051 +0000 UTC m=+192.981354434"
Oct 30 00:35:03.556253 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:03.556215    4393 generic.go:356] "Generic (PLEG): container finished" podID="b4dcc75d-f832-4d5d-86ae-baade25bacd1" containerID="90b684f6785871f295969d983cd61340cf9ed4d014d336261f295f0d631aef93" exitCode=0
Oct 30 00:35:03.556649 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:03.556302    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgdm2f" event={"ID":"b4dcc75d-f832-4d5d-86ae-baade25bacd1","Type":"ContainerDied","Data":"90b684f6785871f295969d983cd61340cf9ed4d014d336261f295f0d631aef93"}
Oct 30 00:35:04.674039 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:04.674012    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgdm2f"
Oct 30 00:35:04.849110 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:04.849034    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4dcc75d-f832-4d5d-86ae-baade25bacd1-kube-api-access-92nv9\" (UniqueName: \"kubernetes.io/projected/b4dcc75d-f832-4d5d-86ae-baade25bacd1-kube-api-access-92nv9\") pod \"b4dcc75d-f832-4d5d-86ae-baade25bacd1\" (UID: \"b4dcc75d-f832-4d5d-86ae-baade25bacd1\") "
Oct 30 00:35:04.849110 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:04.849100    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4dcc75d-f832-4d5d-86ae-baade25bacd1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4dcc75d-f832-4d5d-86ae-baade25bacd1-bundle\") pod \"b4dcc75d-f832-4d5d-86ae-baade25bacd1\" (UID: \"b4dcc75d-f832-4d5d-86ae-baade25bacd1\") "
Oct 30 00:35:04.849300 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:04.849146    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4dcc75d-f832-4d5d-86ae-baade25bacd1-util\" (UniqueName: \"kubernetes.io/empty-dir/b4dcc75d-f832-4d5d-86ae-baade25bacd1-util\") pod \"b4dcc75d-f832-4d5d-86ae-baade25bacd1\" (UID: \"b4dcc75d-f832-4d5d-86ae-baade25bacd1\") "
Oct 30 00:35:04.849803 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:04.849756    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4dcc75d-f832-4d5d-86ae-baade25bacd1-bundle" pod "b4dcc75d-f832-4d5d-86ae-baade25bacd1" (UID: "b4dcc75d-f832-4d5d-86ae-baade25bacd1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:35:04.851398 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:04.851378    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4dcc75d-f832-4d5d-86ae-baade25bacd1-kube-api-access-92nv9" pod "b4dcc75d-f832-4d5d-86ae-baade25bacd1" (UID: "b4dcc75d-f832-4d5d-86ae-baade25bacd1"). InnerVolumeSpecName "kube-api-access-92nv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:35:04.853117 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:04.853088    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4dcc75d-f832-4d5d-86ae-baade25bacd1-util" pod "b4dcc75d-f832-4d5d-86ae-baade25bacd1" (UID: "b4dcc75d-f832-4d5d-86ae-baade25bacd1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:35:04.950106 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:04.950063    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92nv9\" (UniqueName: \"kubernetes.io/projected/b4dcc75d-f832-4d5d-86ae-baade25bacd1-kube-api-access-92nv9\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:35:04.950106 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:04.950098    4393 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4dcc75d-f832-4d5d-86ae-baade25bacd1-bundle\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:35:04.950106 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:04.950108    4393 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4dcc75d-f832-4d5d-86ae-baade25bacd1-util\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:35:05.565074 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:05.565040    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgdm2f" event={"ID":"b4dcc75d-f832-4d5d-86ae-baade25bacd1","Type":"ContainerDied","Data":"9071da14889eadee34f212e762204e1815d3780277c3b92bb9846499c9939a9a"}
Oct 30 00:35:05.565242 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:05.565087    4393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9071da14889eadee34f212e762204e1815d3780277c3b92bb9846499c9939a9a"
Oct 30 00:35:05.565242 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:05.565108    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgdm2f"
Oct 30 00:35:09.943138 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:09.943102    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-k7pjg"]
Oct 30 00:35:09.943707 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:09.943512    4393 cpu_manager.go:414] "containerMap: removing container" podUID="b4dcc75d-f832-4d5d-86ae-baade25bacd1" containerName="extract"
Oct 30 00:35:09.943707 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:09.943528    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4dcc75d-f832-4d5d-86ae-baade25bacd1" containerName="extract"
Oct 30 00:35:09.943707 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:09.943551    4393 cpu_manager.go:414] "containerMap: removing container" podUID="b4dcc75d-f832-4d5d-86ae-baade25bacd1" containerName="pull"
Oct 30 00:35:09.943707 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:09.943561    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4dcc75d-f832-4d5d-86ae-baade25bacd1" containerName="pull"
Oct 30 00:35:09.943707 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:09.943590    4393 cpu_manager.go:414] "containerMap: removing container" podUID="b4dcc75d-f832-4d5d-86ae-baade25bacd1" containerName="util"
Oct 30 00:35:09.943707 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:09.943598    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4dcc75d-f832-4d5d-86ae-baade25bacd1" containerName="util"
Oct 30 00:35:09.943707 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:09.943672    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4dcc75d-f832-4d5d-86ae-baade25bacd1" containerName="extract"
Oct 30 00:35:09.946958 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:09.946938    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-k7pjg"
Oct 30 00:35:09.972667 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:09.972638    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:35:09.972667 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:09.972663    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-gdbww\""
Oct 30 00:35:09.972836 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:09.972653    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 30 00:35:09.973954 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:09.973934    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-k7pjg"]
Oct 30 00:35:10.089926 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:10.089892    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c4eb53f-1bb4-4ae1-8458-2a8ef6513365-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-k7pjg\" (UID: \"4c4eb53f-1bb4-4ae1-8458-2a8ef6513365\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-k7pjg"
Oct 30 00:35:10.090090 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:10.089933    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv7t5\" (UniqueName: \"kubernetes.io/projected/4c4eb53f-1bb4-4ae1-8458-2a8ef6513365-kube-api-access-xv7t5\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-k7pjg\" (UID: \"4c4eb53f-1bb4-4ae1-8458-2a8ef6513365\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-k7pjg"
Oct 30 00:35:10.190491 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:10.190457    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c4eb53f-1bb4-4ae1-8458-2a8ef6513365-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-k7pjg\" (UID: \"4c4eb53f-1bb4-4ae1-8458-2a8ef6513365\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-k7pjg"
Oct 30 00:35:10.190491 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:10.190495    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv7t5\" (UniqueName: \"kubernetes.io/projected/4c4eb53f-1bb4-4ae1-8458-2a8ef6513365-kube-api-access-xv7t5\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-k7pjg\" (UID: \"4c4eb53f-1bb4-4ae1-8458-2a8ef6513365\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-k7pjg"
Oct 30 00:35:10.190893 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:10.190872    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c4eb53f-1bb4-4ae1-8458-2a8ef6513365-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-k7pjg\" (UID: \"4c4eb53f-1bb4-4ae1-8458-2a8ef6513365\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-k7pjg"
Oct 30 00:35:10.214381 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:10.214316    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv7t5\" (UniqueName: \"kubernetes.io/projected/4c4eb53f-1bb4-4ae1-8458-2a8ef6513365-kube-api-access-xv7t5\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-k7pjg\" (UID: \"4c4eb53f-1bb4-4ae1-8458-2a8ef6513365\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-k7pjg"
Oct 30 00:35:10.257488 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:10.257459    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-k7pjg"
Oct 30 00:35:10.580144 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:10.579941    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-k7pjg"]
Oct 30 00:35:10.583155 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:35:10.583129    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c4eb53f_1bb4_4ae1_8458_2a8ef6513365.slice/crio-bda096ee64347c68b00a77d04bcfc7579fe467400a179e1f8c82530297389a15 WatchSource:0}: Error finding container bda096ee64347c68b00a77d04bcfc7579fe467400a179e1f8c82530297389a15: Status 404 returned error can't find the container with id bda096ee64347c68b00a77d04bcfc7579fe467400a179e1f8c82530297389a15
Oct 30 00:35:11.589516 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:11.589480    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-k7pjg" event={"ID":"4c4eb53f-1bb4-4ae1-8458-2a8ef6513365","Type":"ContainerStarted","Data":"bda096ee64347c68b00a77d04bcfc7579fe467400a179e1f8c82530297389a15"}
Oct 30 00:35:12.595592 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:12.595500    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-k7pjg" event={"ID":"4c4eb53f-1bb4-4ae1-8458-2a8ef6513365","Type":"ContainerStarted","Data":"1c37039564cc0b0570afe50d77e2666aaf3f05db0991c0501599c493d68ed758"}
Oct 30 00:35:12.665122 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:12.665078    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-k7pjg" podStartSLOduration=2.012402249 podStartE2EDuration="3.665064893s" totalImagesPullingTime="1.652662644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:35:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:35:12.652737208 +0000 UTC m=+203.040413596" watchObservedRunningTime="2026-10-30 00:35:12.665064893 +0000 UTC m=+203.052741281"
Oct 30 00:35:14.151203 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:14.151173    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mfqzd"]
Oct 30 00:35:14.154637 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:14.154620    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mfqzd"
Oct 30 00:35:14.183505 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:14.183475    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 00:35:14.183505 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:14.183490    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 00:35:14.183711 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:14.183564    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"61b1ff68c7255ee5c5329fe19a5d96293a9578-ce91a85d-dockercfg-cxqrt\""
Oct 30 00:35:14.185445 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:14.185421    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mfqzd"]
Oct 30 00:35:14.223238 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:14.223200    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ddd98fc1-5bb4-4e9c-a037-39b01547533a-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mfqzd\" (UID: \"ddd98fc1-5bb4-4e9c-a037-39b01547533a\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mfqzd"
Oct 30 00:35:14.223435 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:14.223254    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ddd98fc1-5bb4-4e9c-a037-39b01547533a-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mfqzd\" (UID: \"ddd98fc1-5bb4-4e9c-a037-39b01547533a\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mfqzd"
Oct 30 00:35:14.223435 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:14.223377    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9d5r\" (UniqueName: \"kubernetes.io/projected/ddd98fc1-5bb4-4e9c-a037-39b01547533a-kube-api-access-r9d5r\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mfqzd\" (UID: \"ddd98fc1-5bb4-4e9c-a037-39b01547533a\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mfqzd"
Oct 30 00:35:14.323894 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:14.323856    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ddd98fc1-5bb4-4e9c-a037-39b01547533a-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mfqzd\" (UID: \"ddd98fc1-5bb4-4e9c-a037-39b01547533a\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mfqzd"
Oct 30 00:35:14.324064 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:14.323920    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9d5r\" (UniqueName: \"kubernetes.io/projected/ddd98fc1-5bb4-4e9c-a037-39b01547533a-kube-api-access-r9d5r\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mfqzd\" (UID: \"ddd98fc1-5bb4-4e9c-a037-39b01547533a\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mfqzd"
Oct 30 00:35:14.324064 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:14.323956    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ddd98fc1-5bb4-4e9c-a037-39b01547533a-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mfqzd\" (UID: \"ddd98fc1-5bb4-4e9c-a037-39b01547533a\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mfqzd"
Oct 30 00:35:14.324270 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:14.324250    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ddd98fc1-5bb4-4e9c-a037-39b01547533a-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mfqzd\" (UID: \"ddd98fc1-5bb4-4e9c-a037-39b01547533a\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mfqzd"
Oct 30 00:35:14.324325 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:14.324285    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ddd98fc1-5bb4-4e9c-a037-39b01547533a-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mfqzd\" (UID: \"ddd98fc1-5bb4-4e9c-a037-39b01547533a\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mfqzd"
Oct 30 00:35:14.347151 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:14.347118    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9d5r\" (UniqueName: \"kubernetes.io/projected/ddd98fc1-5bb4-4e9c-a037-39b01547533a-kube-api-access-r9d5r\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mfqzd\" (UID: \"ddd98fc1-5bb4-4e9c-a037-39b01547533a\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mfqzd"
Oct 30 00:35:14.469935 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:14.469861    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mfqzd"
Oct 30 00:35:14.765105 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:14.765036    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mfqzd"]
Oct 30 00:35:14.768025 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:35:14.767996    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd98fc1_5bb4_4e9c_a037_39b01547533a.slice/crio-9d250bd72c0c8ef04458d7b23898d527a89e2d271b1b610941ad3ea008a46472 WatchSource:0}: Error finding container 9d250bd72c0c8ef04458d7b23898d527a89e2d271b1b610941ad3ea008a46472: Status 404 returned error can't find the container with id 9d250bd72c0c8ef04458d7b23898d527a89e2d271b1b610941ad3ea008a46472
Oct 30 00:35:15.609813 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:15.609710    4393 generic.go:356] "Generic (PLEG): container finished" podID="ddd98fc1-5bb4-4e9c-a037-39b01547533a" containerID="54a886bb4f215cdbe37407d049f11ba298340f9c0ca97cc8eb7eede567e29470" exitCode=0
Oct 30 00:35:15.609813 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:15.609802    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mfqzd" event={"ID":"ddd98fc1-5bb4-4e9c-a037-39b01547533a","Type":"ContainerDied","Data":"54a886bb4f215cdbe37407d049f11ba298340f9c0ca97cc8eb7eede567e29470"}
Oct 30 00:35:15.610342 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:15.609852    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mfqzd" event={"ID":"ddd98fc1-5bb4-4e9c-a037-39b01547533a","Type":"ContainerStarted","Data":"9d250bd72c0c8ef04458d7b23898d527a89e2d271b1b610941ad3ea008a46472"}
Oct 30 00:35:18.623758 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:18.623723    4393 generic.go:356] "Generic (PLEG): container finished" podID="ddd98fc1-5bb4-4e9c-a037-39b01547533a" containerID="d5a20976e56bd34bbf714bc9148a8d303d2fd944bd23542ab41b9e7fee443b26" exitCode=0
Oct 30 00:35:18.624150 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:18.623817    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mfqzd" event={"ID":"ddd98fc1-5bb4-4e9c-a037-39b01547533a","Type":"ContainerDied","Data":"d5a20976e56bd34bbf714bc9148a8d303d2fd944bd23542ab41b9e7fee443b26"}
Oct 30 00:35:18.656840 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:18.656814    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-7xbxh"]
Oct 30 00:35:18.661204 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:18.661189    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7xbxh"
Oct 30 00:35:18.677773 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:18.677749    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 00:35:18.677981 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:18.677955    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 00:35:18.678141 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:18.678110    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-8qrmk\""
Oct 30 00:35:18.678741 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:18.678723    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-7xbxh"]
Oct 30 00:35:18.758472 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:18.758432    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40039447-b7f8-4d84-ac44-53fb96718eaa-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-7xbxh\" (UID: \"40039447-b7f8-4d84-ac44-53fb96718eaa\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7xbxh"
Oct 30 00:35:18.758616 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:18.758491    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2qvm\" (UniqueName: \"kubernetes.io/projected/40039447-b7f8-4d84-ac44-53fb96718eaa-kube-api-access-b2qvm\") pod \"cert-manager-cainjector-5c57bbd4fc-7xbxh\" (UID: \"40039447-b7f8-4d84-ac44-53fb96718eaa\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7xbxh"
Oct 30 00:35:18.859829 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:18.859798    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2qvm\" (UniqueName: \"kubernetes.io/projected/40039447-b7f8-4d84-ac44-53fb96718eaa-kube-api-access-b2qvm\") pod \"cert-manager-cainjector-5c57bbd4fc-7xbxh\" (UID: \"40039447-b7f8-4d84-ac44-53fb96718eaa\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7xbxh"
Oct 30 00:35:18.859944 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:18.859863    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40039447-b7f8-4d84-ac44-53fb96718eaa-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-7xbxh\" (UID: \"40039447-b7f8-4d84-ac44-53fb96718eaa\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7xbxh"
Oct 30 00:35:18.882352 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:18.882294    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/40039447-b7f8-4d84-ac44-53fb96718eaa-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-7xbxh\" (UID: \"40039447-b7f8-4d84-ac44-53fb96718eaa\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7xbxh"
Oct 30 00:35:18.882446 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:18.882352    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2qvm\" (UniqueName: \"kubernetes.io/projected/40039447-b7f8-4d84-ac44-53fb96718eaa-kube-api-access-b2qvm\") pod \"cert-manager-cainjector-5c57bbd4fc-7xbxh\" (UID: \"40039447-b7f8-4d84-ac44-53fb96718eaa\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7xbxh"
Oct 30 00:35:19.007135 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:19.007104    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7xbxh"
Oct 30 00:35:19.316458 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:19.316437    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-7xbxh"]
Oct 30 00:35:19.318456 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:35:19.318429    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40039447_b7f8_4d84_ac44_53fb96718eaa.slice/crio-5d98ef331b952a4eaebb05988461b17a4d9b227eda81ae939a21547f040416d8 WatchSource:0}: Error finding container 5d98ef331b952a4eaebb05988461b17a4d9b227eda81ae939a21547f040416d8: Status 404 returned error can't find the container with id 5d98ef331b952a4eaebb05988461b17a4d9b227eda81ae939a21547f040416d8
Oct 30 00:35:19.629563 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:19.629531    4393 generic.go:356] "Generic (PLEG): container finished" podID="ddd98fc1-5bb4-4e9c-a037-39b01547533a" containerID="07983a57726eb787b200a25a0b4dc17e384f2bae235a9a8d7469ed812d0ce124" exitCode=0
Oct 30 00:35:19.630012 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:19.629619    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mfqzd" event={"ID":"ddd98fc1-5bb4-4e9c-a037-39b01547533a","Type":"ContainerDied","Data":"07983a57726eb787b200a25a0b4dc17e384f2bae235a9a8d7469ed812d0ce124"}
Oct 30 00:35:19.630772 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:19.630750    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7xbxh" event={"ID":"40039447-b7f8-4d84-ac44-53fb96718eaa","Type":"ContainerStarted","Data":"5d98ef331b952a4eaebb05988461b17a4d9b227eda81ae939a21547f040416d8"}
Oct 30 00:35:20.771418 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:20.771388    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mfqzd"
Oct 30 00:35:20.877631 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:20.877599    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddd98fc1-5bb4-4e9c-a037-39b01547533a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ddd98fc1-5bb4-4e9c-a037-39b01547533a-bundle\") pod \"ddd98fc1-5bb4-4e9c-a037-39b01547533a\" (UID: \"ddd98fc1-5bb4-4e9c-a037-39b01547533a\") "
Oct 30 00:35:20.877832 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:20.877695    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddd98fc1-5bb4-4e9c-a037-39b01547533a-kube-api-access-r9d5r\" (UniqueName: \"kubernetes.io/projected/ddd98fc1-5bb4-4e9c-a037-39b01547533a-kube-api-access-r9d5r\") pod \"ddd98fc1-5bb4-4e9c-a037-39b01547533a\" (UID: \"ddd98fc1-5bb4-4e9c-a037-39b01547533a\") "
Oct 30 00:35:20.877832 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:20.877765    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddd98fc1-5bb4-4e9c-a037-39b01547533a-util\" (UniqueName: \"kubernetes.io/empty-dir/ddd98fc1-5bb4-4e9c-a037-39b01547533a-util\") pod \"ddd98fc1-5bb4-4e9c-a037-39b01547533a\" (UID: \"ddd98fc1-5bb4-4e9c-a037-39b01547533a\") "
Oct 30 00:35:20.878095 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:20.878065    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd98fc1-5bb4-4e9c-a037-39b01547533a-bundle" pod "ddd98fc1-5bb4-4e9c-a037-39b01547533a" (UID: "ddd98fc1-5bb4-4e9c-a037-39b01547533a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:35:20.880368 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:20.880326    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd98fc1-5bb4-4e9c-a037-39b01547533a-kube-api-access-r9d5r" pod "ddd98fc1-5bb4-4e9c-a037-39b01547533a" (UID: "ddd98fc1-5bb4-4e9c-a037-39b01547533a"). InnerVolumeSpecName "kube-api-access-r9d5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:35:20.883481 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:20.883442    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd98fc1-5bb4-4e9c-a037-39b01547533a-util" pod "ddd98fc1-5bb4-4e9c-a037-39b01547533a" (UID: "ddd98fc1-5bb4-4e9c-a037-39b01547533a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:35:20.978963 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:20.978880    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9d5r\" (UniqueName: \"kubernetes.io/projected/ddd98fc1-5bb4-4e9c-a037-39b01547533a-kube-api-access-r9d5r\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:35:20.978963 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:20.978916    4393 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ddd98fc1-5bb4-4e9c-a037-39b01547533a-util\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:35:20.978963 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:20.978930    4393 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ddd98fc1-5bb4-4e9c-a037-39b01547533a-bundle\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:35:21.641655 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:21.641626    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mfqzd"
Oct 30 00:35:21.641655 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:21.641636    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85mfqzd" event={"ID":"ddd98fc1-5bb4-4e9c-a037-39b01547533a","Type":"ContainerDied","Data":"9d250bd72c0c8ef04458d7b23898d527a89e2d271b1b610941ad3ea008a46472"}
Oct 30 00:35:21.641921 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:21.641680    4393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d250bd72c0c8ef04458d7b23898d527a89e2d271b1b610941ad3ea008a46472"
Oct 30 00:35:22.647565 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:22.647533    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7xbxh" event={"ID":"40039447-b7f8-4d84-ac44-53fb96718eaa","Type":"ContainerStarted","Data":"06eb03af87073c5f95576d04eb15ee75f87143300b457c7cba7c8e5c85a8985c"}
Oct 30 00:35:22.687075 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:22.687022    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-7xbxh" podStartSLOduration=2.304829594 podStartE2EDuration="4.687007403s" totalImagesPullingTime="2.382177809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:35:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:35:22.68553694 +0000 UTC m=+213.073213339" watchObservedRunningTime="2026-10-30 00:35:22.687007403 +0000 UTC m=+213.074683792"
Oct 30 00:35:26.509931 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:35:26.509896    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:35:32.135637 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:32.135557    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-5wfhb"]
Oct 30 00:35:32.136031 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:32.135872    4393 cpu_manager.go:414] "containerMap: removing container" podUID="ddd98fc1-5bb4-4e9c-a037-39b01547533a" containerName="util"
Oct 30 00:35:32.136031 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:32.135884    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddd98fc1-5bb4-4e9c-a037-39b01547533a" containerName="util"
Oct 30 00:35:32.136031 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:32.135897    4393 cpu_manager.go:414] "containerMap: removing container" podUID="ddd98fc1-5bb4-4e9c-a037-39b01547533a" containerName="extract"
Oct 30 00:35:32.136031 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:32.135902    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddd98fc1-5bb4-4e9c-a037-39b01547533a" containerName="extract"
Oct 30 00:35:32.136031 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:32.135914    4393 cpu_manager.go:414] "containerMap: removing container" podUID="ddd98fc1-5bb4-4e9c-a037-39b01547533a" containerName="pull"
Oct 30 00:35:32.136031 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:32.135921    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddd98fc1-5bb4-4e9c-a037-39b01547533a" containerName="pull"
Oct 30 00:35:32.136031 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:32.135969    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddd98fc1-5bb4-4e9c-a037-39b01547533a" containerName="extract"
Oct 30 00:35:32.151494 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:32.151473    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-5wfhb"
Oct 30 00:35:32.160647 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:32.160619    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-5wfhb"]
Oct 30 00:35:32.170101 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:32.170079    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 30 00:35:32.170316 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:32.170295    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 30 00:35:32.170357 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:32.170325    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-7zgdk\""
Oct 30 00:35:32.272581 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:32.272545    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phzmx\" (UniqueName: \"kubernetes.io/projected/12890167-4bb7-45d2-9117-85365f3bfbc4-kube-api-access-phzmx\") pod \"openshift-lws-operator-8465994787-5wfhb\" (UID: \"12890167-4bb7-45d2-9117-85365f3bfbc4\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-5wfhb"
Oct 30 00:35:32.272745 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:32.272662    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/12890167-4bb7-45d2-9117-85365f3bfbc4-tmp\") pod \"openshift-lws-operator-8465994787-5wfhb\" (UID: \"12890167-4bb7-45d2-9117-85365f3bfbc4\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-5wfhb"
Oct 30 00:35:32.374079 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:32.374035    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/12890167-4bb7-45d2-9117-85365f3bfbc4-tmp\") pod \"openshift-lws-operator-8465994787-5wfhb\" (UID: \"12890167-4bb7-45d2-9117-85365f3bfbc4\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-5wfhb"
Oct 30 00:35:32.374250 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:32.374101    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phzmx\" (UniqueName: \"kubernetes.io/projected/12890167-4bb7-45d2-9117-85365f3bfbc4-kube-api-access-phzmx\") pod \"openshift-lws-operator-8465994787-5wfhb\" (UID: \"12890167-4bb7-45d2-9117-85365f3bfbc4\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-5wfhb"
Oct 30 00:35:32.374429 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:32.374408    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/12890167-4bb7-45d2-9117-85365f3bfbc4-tmp\") pod \"openshift-lws-operator-8465994787-5wfhb\" (UID: \"12890167-4bb7-45d2-9117-85365f3bfbc4\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-5wfhb"
Oct 30 00:35:32.397005 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:32.396936    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phzmx\" (UniqueName: \"kubernetes.io/projected/12890167-4bb7-45d2-9117-85365f3bfbc4-kube-api-access-phzmx\") pod \"openshift-lws-operator-8465994787-5wfhb\" (UID: \"12890167-4bb7-45d2-9117-85365f3bfbc4\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-5wfhb"
Oct 30 00:35:32.474568 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:32.474525    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-5wfhb"
Oct 30 00:35:32.789460 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:32.789438    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-5wfhb"]
Oct 30 00:35:32.791601 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:35:32.791576    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12890167_4bb7_45d2_9117_85365f3bfbc4.slice/crio-990b68ea7ee5d80e6a1052cd03845c5da957f45a8e40dfb9f2925cdf4ee4a212 WatchSource:0}: Error finding container 990b68ea7ee5d80e6a1052cd03845c5da957f45a8e40dfb9f2925cdf4ee4a212: Status 404 returned error can't find the container with id 990b68ea7ee5d80e6a1052cd03845c5da957f45a8e40dfb9f2925cdf4ee4a212
Oct 30 00:35:33.693015 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:33.692977    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-5wfhb" event={"ID":"12890167-4bb7-45d2-9117-85365f3bfbc4","Type":"ContainerStarted","Data":"990b68ea7ee5d80e6a1052cd03845c5da957f45a8e40dfb9f2925cdf4ee4a212"}
Oct 30 00:35:34.699106 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:34.699074    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-5wfhb" event={"ID":"12890167-4bb7-45d2-9117-85365f3bfbc4","Type":"ContainerStarted","Data":"384f6e896ad95970c2d5b8edd1f6a221e4a8666435e51fe9f241ac04bf1c4150"}
Oct 30 00:35:34.747083 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:34.747033    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-8465994787-5wfhb" podStartSLOduration=0.930020962 podStartE2EDuration="2.747018768s" totalImagesPullingTime="1.816997806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:35:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:35:34.744762449 +0000 UTC m=+225.132438836" watchObservedRunningTime="2026-10-30 00:35:34.747018768 +0000 UTC m=+225.134695156"
Oct 30 00:35:39.135810 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:39.135758    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rhszq"]
Oct 30 00:35:39.138388 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:39.138371    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rhszq"
Oct 30 00:35:39.164146 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:39.164115    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 00:35:39.164146 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:39.164136    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 00:35:39.164373 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:39.164124    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-l8jcd\""
Oct 30 00:35:39.165584 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:39.165567    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rhszq"]
Oct 30 00:35:39.233107 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:39.233070    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b86a6cab-cd47-4450-8056-1240f3277504-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rhszq\" (UID: \"b86a6cab-cd47-4450-8056-1240f3277504\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rhszq"
Oct 30 00:35:39.233107 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:39.233114    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b86a6cab-cd47-4450-8056-1240f3277504-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rhszq\" (UID: \"b86a6cab-cd47-4450-8056-1240f3277504\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rhszq"
Oct 30 00:35:39.233326 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:39.233232    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qftsk\" (UniqueName: \"kubernetes.io/projected/b86a6cab-cd47-4450-8056-1240f3277504-kube-api-access-qftsk\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rhszq\" (UID: \"b86a6cab-cd47-4450-8056-1240f3277504\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rhszq"
Oct 30 00:35:39.333874 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:39.333835    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qftsk\" (UniqueName: \"kubernetes.io/projected/b86a6cab-cd47-4450-8056-1240f3277504-kube-api-access-qftsk\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rhszq\" (UID: \"b86a6cab-cd47-4450-8056-1240f3277504\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rhszq"
Oct 30 00:35:39.333874 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:39.333879    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b86a6cab-cd47-4450-8056-1240f3277504-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rhszq\" (UID: \"b86a6cab-cd47-4450-8056-1240f3277504\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rhszq"
Oct 30 00:35:39.334077 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:39.334005    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b86a6cab-cd47-4450-8056-1240f3277504-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rhszq\" (UID: \"b86a6cab-cd47-4450-8056-1240f3277504\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rhszq"
Oct 30 00:35:39.334197 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:39.334182    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b86a6cab-cd47-4450-8056-1240f3277504-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rhszq\" (UID: \"b86a6cab-cd47-4450-8056-1240f3277504\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rhszq"
Oct 30 00:35:39.334311 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:39.334293    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b86a6cab-cd47-4450-8056-1240f3277504-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rhszq\" (UID: \"b86a6cab-cd47-4450-8056-1240f3277504\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rhszq"
Oct 30 00:35:39.356765 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:39.356734    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qftsk\" (UniqueName: \"kubernetes.io/projected/b86a6cab-cd47-4450-8056-1240f3277504-kube-api-access-qftsk\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rhszq\" (UID: \"b86a6cab-cd47-4450-8056-1240f3277504\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rhszq"
Oct 30 00:35:39.450738 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:39.450673    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rhszq"
Oct 30 00:35:39.743373 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:39.743351    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rhszq"]
Oct 30 00:35:39.746069 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:35:39.746043    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb86a6cab_cd47_4450_8056_1240f3277504.slice/crio-449c58deade9dff70bdc6c774f65dc0ee8bf97802fee3d7248c0ff527a128363 WatchSource:0}: Error finding container 449c58deade9dff70bdc6c774f65dc0ee8bf97802fee3d7248c0ff527a128363: Status 404 returned error can't find the container with id 449c58deade9dff70bdc6c774f65dc0ee8bf97802fee3d7248c0ff527a128363
Oct 30 00:35:40.726489 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:40.726448    4393 generic.go:356] "Generic (PLEG): container finished" podID="b86a6cab-cd47-4450-8056-1240f3277504" containerID="636d5aee06bcfc14980433e519cd07449ee528edc8992cada5c10802fdcd8e60" exitCode=0
Oct 30 00:35:40.726971 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:40.726534    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rhszq" event={"ID":"b86a6cab-cd47-4450-8056-1240f3277504","Type":"ContainerDied","Data":"636d5aee06bcfc14980433e519cd07449ee528edc8992cada5c10802fdcd8e60"}
Oct 30 00:35:40.726971 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:40.726608    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rhszq" event={"ID":"b86a6cab-cd47-4450-8056-1240f3277504","Type":"ContainerStarted","Data":"449c58deade9dff70bdc6c774f65dc0ee8bf97802fee3d7248c0ff527a128363"}
Oct 30 00:35:41.732189 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:41.732159    4393 generic.go:356] "Generic (PLEG): container finished" podID="b86a6cab-cd47-4450-8056-1240f3277504" containerID="ec25704728f1a6fb64c4c077ba1ef59b10e9409a1242f0c11ba6f3fb176cb46a" exitCode=0
Oct 30 00:35:41.732534 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:41.732239    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rhszq" event={"ID":"b86a6cab-cd47-4450-8056-1240f3277504","Type":"ContainerDied","Data":"ec25704728f1a6fb64c4c077ba1ef59b10e9409a1242f0c11ba6f3fb176cb46a"}
Oct 30 00:35:42.738695 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:42.738666    4393 generic.go:356] "Generic (PLEG): container finished" podID="b86a6cab-cd47-4450-8056-1240f3277504" containerID="d62179161033c152abd2e444cfdfbc618f6d9a5f90170001c6a56755e7b6ab41" exitCode=0
Oct 30 00:35:42.739108 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:42.738754    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rhszq" event={"ID":"b86a6cab-cd47-4450-8056-1240f3277504","Type":"ContainerDied","Data":"d62179161033c152abd2e444cfdfbc618f6d9a5f90170001c6a56755e7b6ab41"}
Oct 30 00:35:43.857682 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:43.857658    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rhszq"
Oct 30 00:35:43.971399 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:43.971368    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b86a6cab-cd47-4450-8056-1240f3277504-kube-api-access-qftsk\" (UniqueName: \"kubernetes.io/projected/b86a6cab-cd47-4450-8056-1240f3277504-kube-api-access-qftsk\") pod \"b86a6cab-cd47-4450-8056-1240f3277504\" (UID: \"b86a6cab-cd47-4450-8056-1240f3277504\") "
Oct 30 00:35:43.971576 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:43.971426    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b86a6cab-cd47-4450-8056-1240f3277504-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b86a6cab-cd47-4450-8056-1240f3277504-bundle\") pod \"b86a6cab-cd47-4450-8056-1240f3277504\" (UID: \"b86a6cab-cd47-4450-8056-1240f3277504\") "
Oct 30 00:35:43.971576 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:43.971449    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b86a6cab-cd47-4450-8056-1240f3277504-util\" (UniqueName: \"kubernetes.io/empty-dir/b86a6cab-cd47-4450-8056-1240f3277504-util\") pod \"b86a6cab-cd47-4450-8056-1240f3277504\" (UID: \"b86a6cab-cd47-4450-8056-1240f3277504\") "
Oct 30 00:35:43.972091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:43.972055    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b86a6cab-cd47-4450-8056-1240f3277504-bundle" pod "b86a6cab-cd47-4450-8056-1240f3277504" (UID: "b86a6cab-cd47-4450-8056-1240f3277504"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:35:43.973625 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:43.973607    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b86a6cab-cd47-4450-8056-1240f3277504-kube-api-access-qftsk" pod "b86a6cab-cd47-4450-8056-1240f3277504" (UID: "b86a6cab-cd47-4450-8056-1240f3277504"). InnerVolumeSpecName "kube-api-access-qftsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:35:43.976658 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:43.976631    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b86a6cab-cd47-4450-8056-1240f3277504-util" pod "b86a6cab-cd47-4450-8056-1240f3277504" (UID: "b86a6cab-cd47-4450-8056-1240f3277504"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:35:44.072176 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:44.072106    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qftsk\" (UniqueName: \"kubernetes.io/projected/b86a6cab-cd47-4450-8056-1240f3277504-kube-api-access-qftsk\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:35:44.072176 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:44.072134    4393 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b86a6cab-cd47-4450-8056-1240f3277504-bundle\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:35:44.072176 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:44.072148    4393 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b86a6cab-cd47-4450-8056-1240f3277504-util\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:35:44.749056 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:44.749024    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rhszq" event={"ID":"b86a6cab-cd47-4450-8056-1240f3277504","Type":"ContainerDied","Data":"449c58deade9dff70bdc6c774f65dc0ee8bf97802fee3d7248c0ff527a128363"}
Oct 30 00:35:44.749216 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:44.749065    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rhszq"
Oct 30 00:35:44.749216 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:44.749069    4393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="449c58deade9dff70bdc6c774f65dc0ee8bf97802fee3d7248c0ff527a128363"
Oct 30 00:35:48.780054 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:48.780021    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bcdb85766-mfpvd"]
Oct 30 00:35:48.780402 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:48.780313    4393 cpu_manager.go:414] "containerMap: removing container" podUID="b86a6cab-cd47-4450-8056-1240f3277504" containerName="pull"
Oct 30 00:35:48.780402 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:48.780323    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b86a6cab-cd47-4450-8056-1240f3277504" containerName="pull"
Oct 30 00:35:48.780402 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:48.780333    4393 cpu_manager.go:414] "containerMap: removing container" podUID="b86a6cab-cd47-4450-8056-1240f3277504" containerName="extract"
Oct 30 00:35:48.780402 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:48.780338    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b86a6cab-cd47-4450-8056-1240f3277504" containerName="extract"
Oct 30 00:35:48.780402 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:48.780354    4393 cpu_manager.go:414] "containerMap: removing container" podUID="b86a6cab-cd47-4450-8056-1240f3277504" containerName="util"
Oct 30 00:35:48.780402 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:48.780358    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b86a6cab-cd47-4450-8056-1240f3277504" containerName="util"
Oct 30 00:35:48.780599 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:48.780419    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="b86a6cab-cd47-4450-8056-1240f3277504" containerName="extract"
Oct 30 00:35:48.782449 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:48.782433    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-mfpvd"
Oct 30 00:35:48.812982 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:48.812955    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 00:35:48.813163 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:48.813145    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 00:35:48.814248 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:48.814224    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bcdb85766-mfpvd"]
Oct 30 00:35:48.819742 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:48.819724    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 00:35:48.820028 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:48.820012    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\""
Oct 30 00:35:48.820260 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:48.820238    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-gqpv5\""
Oct 30 00:35:48.820260 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:48.820242    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 00:35:48.913228 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:48.913193    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bd326a32-6e36-4753-88fd-2855e415cc15-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bcdb85766-mfpvd\" (UID: \"bd326a32-6e36-4753-88fd-2855e415cc15\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-mfpvd"
Oct 30 00:35:48.913228 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:48.913228    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zk9k\" (UniqueName: \"kubernetes.io/projected/bd326a32-6e36-4753-88fd-2855e415cc15-kube-api-access-4zk9k\") pod \"opendatahub-operator-controller-manager-6bcdb85766-mfpvd\" (UID: \"bd326a32-6e36-4753-88fd-2855e415cc15\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-mfpvd"
Oct 30 00:35:48.913426 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:48.913254    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bd326a32-6e36-4753-88fd-2855e415cc15-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bcdb85766-mfpvd\" (UID: \"bd326a32-6e36-4753-88fd-2855e415cc15\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-mfpvd"
Oct 30 00:35:48.913426 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:48.913373    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bd326a32-6e36-4753-88fd-2855e415cc15-manifests\") pod \"opendatahub-operator-controller-manager-6bcdb85766-mfpvd\" (UID: \"bd326a32-6e36-4753-88fd-2855e415cc15\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-mfpvd"
Oct 30 00:35:48.913426 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:48.913419    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd326a32-6e36-4753-88fd-2855e415cc15-tmp\") pod \"opendatahub-operator-controller-manager-6bcdb85766-mfpvd\" (UID: \"bd326a32-6e36-4753-88fd-2855e415cc15\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-mfpvd"
Oct 30 00:35:48.913528 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:48.913508    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bd326a32-6e36-4753-88fd-2855e415cc15-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bcdb85766-mfpvd\" (UID: \"bd326a32-6e36-4753-88fd-2855e415cc15\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-mfpvd"
Oct 30 00:35:49.014164 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:49.014126    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bd326a32-6e36-4753-88fd-2855e415cc15-manifests\") pod \"opendatahub-operator-controller-manager-6bcdb85766-mfpvd\" (UID: \"bd326a32-6e36-4753-88fd-2855e415cc15\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-mfpvd"
Oct 30 00:35:49.014347 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:49.014170    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd326a32-6e36-4753-88fd-2855e415cc15-tmp\") pod \"opendatahub-operator-controller-manager-6bcdb85766-mfpvd\" (UID: \"bd326a32-6e36-4753-88fd-2855e415cc15\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-mfpvd"
Oct 30 00:35:49.014347 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:49.014198    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bd326a32-6e36-4753-88fd-2855e415cc15-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bcdb85766-mfpvd\" (UID: \"bd326a32-6e36-4753-88fd-2855e415cc15\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-mfpvd"
Oct 30 00:35:49.014347 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:49.014222    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bd326a32-6e36-4753-88fd-2855e415cc15-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bcdb85766-mfpvd\" (UID: \"bd326a32-6e36-4753-88fd-2855e415cc15\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-mfpvd"
Oct 30 00:35:49.014347 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:49.014241    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zk9k\" (UniqueName: \"kubernetes.io/projected/bd326a32-6e36-4753-88fd-2855e415cc15-kube-api-access-4zk9k\") pod \"opendatahub-operator-controller-manager-6bcdb85766-mfpvd\" (UID: \"bd326a32-6e36-4753-88fd-2855e415cc15\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-mfpvd"
Oct 30 00:35:49.014347 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:49.014259    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bd326a32-6e36-4753-88fd-2855e415cc15-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bcdb85766-mfpvd\" (UID: \"bd326a32-6e36-4753-88fd-2855e415cc15\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-mfpvd"
Oct 30 00:35:49.014739 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:49.014592    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bd326a32-6e36-4753-88fd-2855e415cc15-manifests\") pod \"opendatahub-operator-controller-manager-6bcdb85766-mfpvd\" (UID: \"bd326a32-6e36-4753-88fd-2855e415cc15\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-mfpvd"
Oct 30 00:35:49.016881 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:49.016848    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bd326a32-6e36-4753-88fd-2855e415cc15-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bcdb85766-mfpvd\" (UID: \"bd326a32-6e36-4753-88fd-2855e415cc15\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-mfpvd"
Oct 30 00:35:49.017000 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:49.016876    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd326a32-6e36-4753-88fd-2855e415cc15-tmp\") pod \"opendatahub-operator-controller-manager-6bcdb85766-mfpvd\" (UID: \"bd326a32-6e36-4753-88fd-2855e415cc15\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-mfpvd"
Oct 30 00:35:49.017000 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:49.016968    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bd326a32-6e36-4753-88fd-2855e415cc15-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bcdb85766-mfpvd\" (UID: \"bd326a32-6e36-4753-88fd-2855e415cc15\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-mfpvd"
Oct 30 00:35:49.017000 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:49.016987    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bd326a32-6e36-4753-88fd-2855e415cc15-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bcdb85766-mfpvd\" (UID: \"bd326a32-6e36-4753-88fd-2855e415cc15\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-mfpvd"
Oct 30 00:35:49.045698 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:49.045642    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zk9k\" (UniqueName: \"kubernetes.io/projected/bd326a32-6e36-4753-88fd-2855e415cc15-kube-api-access-4zk9k\") pod \"opendatahub-operator-controller-manager-6bcdb85766-mfpvd\" (UID: \"bd326a32-6e36-4753-88fd-2855e415cc15\") " pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-mfpvd"
Oct 30 00:35:49.094337 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:49.094309    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-mfpvd"
Oct 30 00:35:49.466012 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:49.465974    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bcdb85766-mfpvd"]
Oct 30 00:35:49.480771 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:35:49.480741    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd326a32_6e36_4753_88fd_2855e415cc15.slice/crio-dfae4643d7dbba62da2852b276653f5077b8d50880971b2404afe085de1b3eab WatchSource:0}: Error finding container dfae4643d7dbba62da2852b276653f5077b8d50880971b2404afe085de1b3eab: Status 404 returned error can't find the container with id dfae4643d7dbba62da2852b276653f5077b8d50880971b2404afe085de1b3eab
Oct 30 00:35:49.768876 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:49.768804    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-mfpvd" event={"ID":"bd326a32-6e36-4753-88fd-2855e415cc15","Type":"ContainerStarted","Data":"dfae4643d7dbba62da2852b276653f5077b8d50880971b2404afe085de1b3eab"}
Oct 30 00:35:52.785548 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:52.785514    4393 generic.go:356] "Generic (PLEG): container finished" podID="bd326a32-6e36-4753-88fd-2855e415cc15" containerID="f771557218a0fe9009df47bbac9c66a5f56006db59cfcf1f531375d4b37fbc26" exitCode=0
Oct 30 00:35:52.785970 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:52.785552    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-mfpvd" event={"ID":"bd326a32-6e36-4753-88fd-2855e415cc15","Type":"ContainerDied","Data":"f771557218a0fe9009df47bbac9c66a5f56006db59cfcf1f531375d4b37fbc26"}
Oct 30 00:35:53.791547 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:53.791511    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-mfpvd" event={"ID":"bd326a32-6e36-4753-88fd-2855e415cc15","Type":"ContainerStarted","Data":"2f9105e6cebcb63eb25f692eface9dbd5e00b2ef7b2defb13de5fc9440e97951"}
Oct 30 00:35:53.791964 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:53.791713    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-mfpvd"
Oct 30 00:35:53.878024 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:35:53.877972    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-mfpvd" podStartSLOduration=2.327752747 podStartE2EDuration="5.877958368s" totalImagesPullingTime="3.112839351s" totalInitContainerRuntime="437.36627ms" isStatefulPod=false podCreationTimestamp="2026-10-30 00:35:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:35:53.866138796 +0000 UTC m=+244.253815188" watchObservedRunningTime="2026-10-30 00:35:53.877958368 +0000 UTC m=+244.265634818"
Oct 30 00:35:56.508873 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:35:56.508838    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:36:04.798404 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:04.798368    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6bcdb85766-mfpvd"
Oct 30 00:36:08.941474 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:08.941434    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-b7df967f4-5hvrq"]
Oct 30 00:36:08.945361 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:08.945336    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b7df967f4-5hvrq"
Oct 30 00:36:08.963616 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:08.963593    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 00:36:08.975372 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:08.975339    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 00:36:08.975372 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:08.975360    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 00:36:08.975585 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:08.975383    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-jd954\""
Oct 30 00:36:08.975585 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:08.975426    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 00:36:08.977015 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:08.976990    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b7df967f4-5hvrq"]
Oct 30 00:36:09.083970 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:09.083938    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq7n8\" (UniqueName: \"kubernetes.io/projected/e1c4b01c-5605-474b-8e98-91f81fe6873d-kube-api-access-mq7n8\") pod \"kube-auth-proxy-b7df967f4-5hvrq\" (UID: \"e1c4b01c-5605-474b-8e98-91f81fe6873d\") " pod="openshift-ingress/kube-auth-proxy-b7df967f4-5hvrq"
Oct 30 00:36:09.083970 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:09.083973    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1c4b01c-5605-474b-8e98-91f81fe6873d-tls-certs\") pod \"kube-auth-proxy-b7df967f4-5hvrq\" (UID: \"e1c4b01c-5605-474b-8e98-91f81fe6873d\") " pod="openshift-ingress/kube-auth-proxy-b7df967f4-5hvrq"
Oct 30 00:36:09.084173 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:09.084107    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1c4b01c-5605-474b-8e98-91f81fe6873d-tmp\") pod \"kube-auth-proxy-b7df967f4-5hvrq\" (UID: \"e1c4b01c-5605-474b-8e98-91f81fe6873d\") " pod="openshift-ingress/kube-auth-proxy-b7df967f4-5hvrq"
Oct 30 00:36:09.185034 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:09.184998    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq7n8\" (UniqueName: \"kubernetes.io/projected/e1c4b01c-5605-474b-8e98-91f81fe6873d-kube-api-access-mq7n8\") pod \"kube-auth-proxy-b7df967f4-5hvrq\" (UID: \"e1c4b01c-5605-474b-8e98-91f81fe6873d\") " pod="openshift-ingress/kube-auth-proxy-b7df967f4-5hvrq"
Oct 30 00:36:09.185034 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:09.185038    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1c4b01c-5605-474b-8e98-91f81fe6873d-tls-certs\") pod \"kube-auth-proxy-b7df967f4-5hvrq\" (UID: \"e1c4b01c-5605-474b-8e98-91f81fe6873d\") " pod="openshift-ingress/kube-auth-proxy-b7df967f4-5hvrq"
Oct 30 00:36:09.185285 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:09.185098    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1c4b01c-5605-474b-8e98-91f81fe6873d-tmp\") pod \"kube-auth-proxy-b7df967f4-5hvrq\" (UID: \"e1c4b01c-5605-474b-8e98-91f81fe6873d\") " pod="openshift-ingress/kube-auth-proxy-b7df967f4-5hvrq"
Oct 30 00:36:09.187597 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:09.187567    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1c4b01c-5605-474b-8e98-91f81fe6873d-tmp\") pod \"kube-auth-proxy-b7df967f4-5hvrq\" (UID: \"e1c4b01c-5605-474b-8e98-91f81fe6873d\") " pod="openshift-ingress/kube-auth-proxy-b7df967f4-5hvrq"
Oct 30 00:36:09.187723 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:09.187704    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1c4b01c-5605-474b-8e98-91f81fe6873d-tls-certs\") pod \"kube-auth-proxy-b7df967f4-5hvrq\" (UID: \"e1c4b01c-5605-474b-8e98-91f81fe6873d\") " pod="openshift-ingress/kube-auth-proxy-b7df967f4-5hvrq"
Oct 30 00:36:09.212363 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:09.212289    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq7n8\" (UniqueName: \"kubernetes.io/projected/e1c4b01c-5605-474b-8e98-91f81fe6873d-kube-api-access-mq7n8\") pod \"kube-auth-proxy-b7df967f4-5hvrq\" (UID: \"e1c4b01c-5605-474b-8e98-91f81fe6873d\") " pod="openshift-ingress/kube-auth-proxy-b7df967f4-5hvrq"
Oct 30 00:36:09.257840 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:09.257805    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b7df967f4-5hvrq"
Oct 30 00:36:09.568263 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:09.568240    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b7df967f4-5hvrq"]
Oct 30 00:36:09.570496 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:36:09.570461    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1c4b01c_5605_474b_8e98_91f81fe6873d.slice/crio-32e3f4a36f7b026d1b1b5978f7e8968afdf247a9994e3200779812f2a8118a27 WatchSource:0}: Error finding container 32e3f4a36f7b026d1b1b5978f7e8968afdf247a9994e3200779812f2a8118a27: Status 404 returned error can't find the container with id 32e3f4a36f7b026d1b1b5978f7e8968afdf247a9994e3200779812f2a8118a27
Oct 30 00:36:09.860222 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:09.860033    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b7df967f4-5hvrq" event={"ID":"e1c4b01c-5605-474b-8e98-91f81fe6873d","Type":"ContainerStarted","Data":"32e3f4a36f7b026d1b1b5978f7e8968afdf247a9994e3200779812f2a8118a27"}
Oct 30 00:36:10.657018 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:10.656981    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g"]
Oct 30 00:36:10.660894 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:10.660467    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g"
Oct 30 00:36:10.675656 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:10.675612    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 00:36:10.675829 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:10.675721    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 00:36:10.675891 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:10.675832    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-k4cpq\""
Oct 30 00:36:10.675891 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:10.675835    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 00:36:10.687940 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:10.687913    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 00:36:10.690070 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:10.690044    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g"]
Oct 30 00:36:10.698442 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:10.698414    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-d695g\" (UID: \"ac007403-e5ac-4a40-b5dd-7cc473f6dd49\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g"
Oct 30 00:36:10.698561 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:10.698453    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-d695g\" (UID: \"ac007403-e5ac-4a40-b5dd-7cc473f6dd49\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g"
Oct 30 00:36:10.698561 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:10.698477    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-d695g\" (UID: \"ac007403-e5ac-4a40-b5dd-7cc473f6dd49\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g"
Oct 30 00:36:10.698677 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:10.698567    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-d695g\" (UID: \"ac007403-e5ac-4a40-b5dd-7cc473f6dd49\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g"
Oct 30 00:36:10.698677 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:10.698614    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-d695g\" (UID: \"ac007403-e5ac-4a40-b5dd-7cc473f6dd49\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g"
Oct 30 00:36:10.698799 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:10.698738    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97smd\" (UniqueName: \"kubernetes.io/projected/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-kube-api-access-97smd\") pod \"istiod-openshift-gateway-7876bfcb74-d695g\" (UID: \"ac007403-e5ac-4a40-b5dd-7cc473f6dd49\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g"
Oct 30 00:36:10.698860 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:10.698808    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-d695g\" (UID: \"ac007403-e5ac-4a40-b5dd-7cc473f6dd49\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g"
Oct 30 00:36:10.799375 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:10.799332    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-d695g\" (UID: \"ac007403-e5ac-4a40-b5dd-7cc473f6dd49\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g"
Oct 30 00:36:10.799567 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:10.799391    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-d695g\" (UID: \"ac007403-e5ac-4a40-b5dd-7cc473f6dd49\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g"
Oct 30 00:36:10.799567 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:10.799427    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97smd\" (UniqueName: \"kubernetes.io/projected/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-kube-api-access-97smd\") pod \"istiod-openshift-gateway-7876bfcb74-d695g\" (UID: \"ac007403-e5ac-4a40-b5dd-7cc473f6dd49\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g"
Oct 30 00:36:10.799567 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:10.799471    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-d695g\" (UID: \"ac007403-e5ac-4a40-b5dd-7cc473f6dd49\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g"
Oct 30 00:36:10.799567 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:10.799517    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-d695g\" (UID: \"ac007403-e5ac-4a40-b5dd-7cc473f6dd49\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g"
Oct 30 00:36:10.799567 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:10.799548    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-d695g\" (UID: \"ac007403-e5ac-4a40-b5dd-7cc473f6dd49\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g"
Oct 30 00:36:10.799876 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:10.799572    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-d695g\" (UID: \"ac007403-e5ac-4a40-b5dd-7cc473f6dd49\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g"
Oct 30 00:36:10.800756 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:10.800728    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-d695g\" (UID: \"ac007403-e5ac-4a40-b5dd-7cc473f6dd49\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g"
Oct 30 00:36:10.802563 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:10.802441    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-d695g\" (UID: \"ac007403-e5ac-4a40-b5dd-7cc473f6dd49\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g"
Oct 30 00:36:10.802719 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:10.802697    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-d695g\" (UID: \"ac007403-e5ac-4a40-b5dd-7cc473f6dd49\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g"
Oct 30 00:36:10.802941 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:10.802919    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-d695g\" (UID: \"ac007403-e5ac-4a40-b5dd-7cc473f6dd49\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g"
Oct 30 00:36:10.803005 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:10.802952    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-d695g\" (UID: \"ac007403-e5ac-4a40-b5dd-7cc473f6dd49\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g"
Oct 30 00:36:10.826082 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:10.826049    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97smd\" (UniqueName: \"kubernetes.io/projected/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-kube-api-access-97smd\") pod \"istiod-openshift-gateway-7876bfcb74-d695g\" (UID: \"ac007403-e5ac-4a40-b5dd-7cc473f6dd49\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g"
Oct 30 00:36:10.838490 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:10.838457    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-d695g\" (UID: \"ac007403-e5ac-4a40-b5dd-7cc473f6dd49\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g"
Oct 30 00:36:10.975293 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:10.975213    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g"
Oct 30 00:36:11.295610 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:11.295534    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g"]
Oct 30 00:36:11.299183 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:36:11.299151    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac007403_e5ac_4a40_b5dd_7cc473f6dd49.slice/crio-1c9fbbe4b984198369ee99cc8787ba037b2dab78454b6666b85b179ec05ea04a WatchSource:0}: Error finding container 1c9fbbe4b984198369ee99cc8787ba037b2dab78454b6666b85b179ec05ea04a: Status 404 returned error can't find the container with id 1c9fbbe4b984198369ee99cc8787ba037b2dab78454b6666b85b179ec05ea04a
Oct 30 00:36:11.871123 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:11.871084    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g" event={"ID":"ac007403-e5ac-4a40-b5dd-7cc473f6dd49","Type":"ContainerStarted","Data":"1c9fbbe4b984198369ee99cc8787ba037b2dab78454b6666b85b179ec05ea04a"}
Oct 30 00:36:12.878770 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:12.878730    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b7df967f4-5hvrq" event={"ID":"e1c4b01c-5605-474b-8e98-91f81fe6873d","Type":"ContainerStarted","Data":"b313c6bf4549c6d1dc7d173fe8adefd38bb3d3dbd2484a6872f1bbd13c0b4e55"}
Oct 30 00:36:12.919443 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:12.919381    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-b7df967f4-5hvrq" podStartSLOduration=2.404818981 podStartE2EDuration="4.919363107s" totalImagesPullingTime="2.514544126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:36:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:36:12.918020248 +0000 UTC m=+263.305696650" watchObservedRunningTime="2026-10-30 00:36:12.919363107 +0000 UTC m=+263.307039496"
Oct 30 00:36:13.950869 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:13.950837    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:36:13.951136 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:13.950900    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:36:14.889364 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:14.889329    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g" event={"ID":"ac007403-e5ac-4a40-b5dd-7cc473f6dd49","Type":"ContainerStarted","Data":"cb807e693f5bde0893e81b3be56a9e497382c3af07fc177f769d9570ce22d822"}
Oct 30 00:36:14.889549 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:14.889457    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g"
Oct 30 00:36:14.961276 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:14.961223    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g" podStartSLOduration=2.313010034 podStartE2EDuration="4.961209245s" totalImagesPullingTime="2.648199211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:36:14.948942347 +0000 UTC m=+265.336618736" watchObservedRunningTime="2026-10-30 00:36:14.961209245 +0000 UTC m=+265.348885634"
Oct 30 00:36:15.568667 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:15.568628    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw"]
Oct 30 00:36:15.573383 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:15.573356    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw"
Oct 30 00:36:15.606842 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:15.606809    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw"]
Oct 30 00:36:15.639372 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:15.639343    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d685e131-7f49-4c0d-870f-1d1a51bd1b74-istio-data\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw\" (UID: \"d685e131-7f49-4c0d-870f-1d1a51bd1b74\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw"
Oct 30 00:36:15.639564 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:15.639388    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d685e131-7f49-4c0d-870f-1d1a51bd1b74-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw\" (UID: \"d685e131-7f49-4c0d-870f-1d1a51bd1b74\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw"
Oct 30 00:36:15.639564 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:15.639423    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgftw\" (UniqueName: \"kubernetes.io/projected/d685e131-7f49-4c0d-870f-1d1a51bd1b74-kube-api-access-zgftw\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw\" (UID: \"d685e131-7f49-4c0d-870f-1d1a51bd1b74\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw"
Oct 30 00:36:15.639564 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:15.639464    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d685e131-7f49-4c0d-870f-1d1a51bd1b74-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw\" (UID: \"d685e131-7f49-4c0d-870f-1d1a51bd1b74\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw"
Oct 30 00:36:15.639564 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:15.639514    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d685e131-7f49-4c0d-870f-1d1a51bd1b74-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw\" (UID: \"d685e131-7f49-4c0d-870f-1d1a51bd1b74\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw"
Oct 30 00:36:15.639564 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:15.639537    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d685e131-7f49-4c0d-870f-1d1a51bd1b74-istio-token\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw\" (UID: \"d685e131-7f49-4c0d-870f-1d1a51bd1b74\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw"
Oct 30 00:36:15.639802 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:15.639596    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d685e131-7f49-4c0d-870f-1d1a51bd1b74-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw\" (UID: \"d685e131-7f49-4c0d-870f-1d1a51bd1b74\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw"
Oct 30 00:36:15.639802 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:15.639631    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d685e131-7f49-4c0d-870f-1d1a51bd1b74-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw\" (UID: \"d685e131-7f49-4c0d-870f-1d1a51bd1b74\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw"
Oct 30 00:36:15.639802 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:15.639682    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d685e131-7f49-4c0d-870f-1d1a51bd1b74-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw\" (UID: \"d685e131-7f49-4c0d-870f-1d1a51bd1b74\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw"
Oct 30 00:36:15.652136 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:15.652102    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-6f2pv\""
Oct 30 00:36:15.740664 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:15.740623    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d685e131-7f49-4c0d-870f-1d1a51bd1b74-istio-data\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw\" (UID: \"d685e131-7f49-4c0d-870f-1d1a51bd1b74\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw"
Oct 30 00:36:15.740664 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:15.740671    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d685e131-7f49-4c0d-870f-1d1a51bd1b74-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw\" (UID: \"d685e131-7f49-4c0d-870f-1d1a51bd1b74\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw"
Oct 30 00:36:15.740932 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:15.740699    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgftw\" (UniqueName: \"kubernetes.io/projected/d685e131-7f49-4c0d-870f-1d1a51bd1b74-kube-api-access-zgftw\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw\" (UID: \"d685e131-7f49-4c0d-870f-1d1a51bd1b74\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw"
Oct 30 00:36:15.740932 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:15.740723    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d685e131-7f49-4c0d-870f-1d1a51bd1b74-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw\" (UID: \"d685e131-7f49-4c0d-870f-1d1a51bd1b74\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw"
Oct 30 00:36:15.740932 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:15.740744    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d685e131-7f49-4c0d-870f-1d1a51bd1b74-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw\" (UID: \"d685e131-7f49-4c0d-870f-1d1a51bd1b74\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw"
Oct 30 00:36:15.740932 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:15.740760    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d685e131-7f49-4c0d-870f-1d1a51bd1b74-istio-token\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw\" (UID: \"d685e131-7f49-4c0d-870f-1d1a51bd1b74\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw"
Oct 30 00:36:15.740932 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:15.740824    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d685e131-7f49-4c0d-870f-1d1a51bd1b74-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw\" (UID: \"d685e131-7f49-4c0d-870f-1d1a51bd1b74\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw"
Oct 30 00:36:15.740932 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:15.740841    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d685e131-7f49-4c0d-870f-1d1a51bd1b74-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw\" (UID: \"d685e131-7f49-4c0d-870f-1d1a51bd1b74\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw"
Oct 30 00:36:15.740932 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:15.740866    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d685e131-7f49-4c0d-870f-1d1a51bd1b74-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw\" (UID: \"d685e131-7f49-4c0d-870f-1d1a51bd1b74\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw"
Oct 30 00:36:15.741313 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:15.741164    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d685e131-7f49-4c0d-870f-1d1a51bd1b74-istio-data\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw\" (UID: \"d685e131-7f49-4c0d-870f-1d1a51bd1b74\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw"
Oct 30 00:36:15.741313 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:15.741174    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d685e131-7f49-4c0d-870f-1d1a51bd1b74-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw\" (UID: \"d685e131-7f49-4c0d-870f-1d1a51bd1b74\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw"
Oct 30 00:36:15.741439 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:15.741359    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d685e131-7f49-4c0d-870f-1d1a51bd1b74-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw\" (UID: \"d685e131-7f49-4c0d-870f-1d1a51bd1b74\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw"
Oct 30 00:36:15.741596 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:15.741572    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d685e131-7f49-4c0d-870f-1d1a51bd1b74-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw\" (UID: \"d685e131-7f49-4c0d-870f-1d1a51bd1b74\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw"
Oct 30 00:36:15.741667 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:15.741579    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d685e131-7f49-4c0d-870f-1d1a51bd1b74-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw\" (UID: \"d685e131-7f49-4c0d-870f-1d1a51bd1b74\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw"
Oct 30 00:36:15.743504 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:15.743487    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d685e131-7f49-4c0d-870f-1d1a51bd1b74-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw\" (UID: \"d685e131-7f49-4c0d-870f-1d1a51bd1b74\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw"
Oct 30 00:36:15.743708 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:15.743691    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d685e131-7f49-4c0d-870f-1d1a51bd1b74-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw\" (UID: \"d685e131-7f49-4c0d-870f-1d1a51bd1b74\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw"
Oct 30 00:36:15.764145 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:15.764121    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d685e131-7f49-4c0d-870f-1d1a51bd1b74-istio-token\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw\" (UID: \"d685e131-7f49-4c0d-870f-1d1a51bd1b74\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw"
Oct 30 00:36:15.764275 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:15.764258    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgftw\" (UniqueName: \"kubernetes.io/projected/d685e131-7f49-4c0d-870f-1d1a51bd1b74-kube-api-access-zgftw\") pod \"data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw\" (UID: \"d685e131-7f49-4c0d-870f-1d1a51bd1b74\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw"
Oct 30 00:36:15.889231 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:15.889196    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw"
Oct 30 00:36:15.896104 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:15.896081    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g"
Oct 30 00:36:16.206622 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:16.206598    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw"]
Oct 30 00:36:16.208794 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:36:16.208747    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd685e131_7f49_4c0d_870f_1d1a51bd1b74.slice/crio-a3d6da607a9f7a9d7ef23702121acd8fefc45edadbdb01b56804e772094cecb2 WatchSource:0}: Error finding container a3d6da607a9f7a9d7ef23702121acd8fefc45edadbdb01b56804e772094cecb2: Status 404 returned error can't find the container with id a3d6da607a9f7a9d7ef23702121acd8fefc45edadbdb01b56804e772094cecb2
Oct 30 00:36:16.899945 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:16.899902    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw" event={"ID":"d685e131-7f49-4c0d-870f-1d1a51bd1b74","Type":"ContainerStarted","Data":"a3d6da607a9f7a9d7ef23702121acd8fefc45edadbdb01b56804e772094cecb2"}
Oct 30 00:36:19.026014 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:19.025972    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:36:19.026326 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:19.026065    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:36:19.026326 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:19.026108    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:36:19.914949 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:19.914912    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw" event={"ID":"d685e131-7f49-4c0d-870f-1d1a51bd1b74","Type":"ContainerStarted","Data":"73f66db55ad4d2556b7d203a01ca4af0f6ece998ccda26b2adad464342b7efb7"}
Oct 30 00:36:19.970751 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:19.970698    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw" podStartSLOduration=2.156745281 podStartE2EDuration="4.970682385s" totalImagesPullingTime="2.813937104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:36:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:36:19.969423421 +0000 UTC m=+270.357099810" watchObservedRunningTime="2026-10-30 00:36:19.970682385 +0000 UTC m=+270.358358775"
Oct 30 00:36:20.889552 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:20.889519    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw"
Oct 30 00:36:20.893859 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:20.893835    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw"
Oct 30 00:36:20.919907 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:20.919875    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw"
Oct 30 00:36:20.920672 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:36:20.920657    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw"
Oct 30 00:36:26.509714 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:36:26.509683    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:36:56.509514 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:36:56.509474    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:37:14.072532 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:14.072489    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4"]
Oct 30 00:37:14.075166 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:14.075137    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4"
Oct 30 00:37:14.104757 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:14.104730    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4"]
Oct 30 00:37:14.164456 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:14.164428    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0479058f-6642-46e3-83a2-2d32949309a1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5nr4\" (UID: \"0479058f-6642-46e3-83a2-2d32949309a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4"
Oct 30 00:37:14.164608 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:14.164462    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb6kp\" (UniqueName: \"kubernetes.io/projected/0479058f-6642-46e3-83a2-2d32949309a1-kube-api-access-fb6kp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5nr4\" (UID: \"0479058f-6642-46e3-83a2-2d32949309a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4"
Oct 30 00:37:14.164649 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:14.164602    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0479058f-6642-46e3-83a2-2d32949309a1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5nr4\" (UID: \"0479058f-6642-46e3-83a2-2d32949309a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4"
Oct 30 00:37:14.164649 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:14.164622    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0479058f-6642-46e3-83a2-2d32949309a1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5nr4\" (UID: \"0479058f-6642-46e3-83a2-2d32949309a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4"
Oct 30 00:37:14.164649 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:14.164641    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0479058f-6642-46e3-83a2-2d32949309a1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5nr4\" (UID: \"0479058f-6642-46e3-83a2-2d32949309a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4"
Oct 30 00:37:14.164747 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:14.164699    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0479058f-6642-46e3-83a2-2d32949309a1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5nr4\" (UID: \"0479058f-6642-46e3-83a2-2d32949309a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4"
Oct 30 00:37:14.164817 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:14.164754    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0479058f-6642-46e3-83a2-2d32949309a1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5nr4\" (UID: \"0479058f-6642-46e3-83a2-2d32949309a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4"
Oct 30 00:37:14.265875 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:14.265843    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0479058f-6642-46e3-83a2-2d32949309a1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5nr4\" (UID: \"0479058f-6642-46e3-83a2-2d32949309a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4"
Oct 30 00:37:14.266037 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:14.265891    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0479058f-6642-46e3-83a2-2d32949309a1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5nr4\" (UID: \"0479058f-6642-46e3-83a2-2d32949309a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4"
Oct 30 00:37:14.266037 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:14.265929    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0479058f-6642-46e3-83a2-2d32949309a1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5nr4\" (UID: \"0479058f-6642-46e3-83a2-2d32949309a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4"
Oct 30 00:37:14.266037 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:14.265959    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0479058f-6642-46e3-83a2-2d32949309a1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5nr4\" (UID: \"0479058f-6642-46e3-83a2-2d32949309a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4"
Oct 30 00:37:14.266202 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:14.266069    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb6kp\" (UniqueName: \"kubernetes.io/projected/0479058f-6642-46e3-83a2-2d32949309a1-kube-api-access-fb6kp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5nr4\" (UID: \"0479058f-6642-46e3-83a2-2d32949309a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4"
Oct 30 00:37:14.266202 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:14.266176    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0479058f-6642-46e3-83a2-2d32949309a1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5nr4\" (UID: \"0479058f-6642-46e3-83a2-2d32949309a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4"
Oct 30 00:37:14.266297 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:14.266201    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0479058f-6642-46e3-83a2-2d32949309a1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5nr4\" (UID: \"0479058f-6642-46e3-83a2-2d32949309a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4"
Oct 30 00:37:14.266772 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:14.266706    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0479058f-6642-46e3-83a2-2d32949309a1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5nr4\" (UID: \"0479058f-6642-46e3-83a2-2d32949309a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4"
Oct 30 00:37:14.268405 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:14.268375    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0479058f-6642-46e3-83a2-2d32949309a1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5nr4\" (UID: \"0479058f-6642-46e3-83a2-2d32949309a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4"
Oct 30 00:37:14.268405 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:14.268399    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0479058f-6642-46e3-83a2-2d32949309a1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5nr4\" (UID: \"0479058f-6642-46e3-83a2-2d32949309a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4"
Oct 30 00:37:14.268562 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:14.268487    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0479058f-6642-46e3-83a2-2d32949309a1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5nr4\" (UID: \"0479058f-6642-46e3-83a2-2d32949309a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4"
Oct 30 00:37:14.268657 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:14.268638    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0479058f-6642-46e3-83a2-2d32949309a1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5nr4\" (UID: \"0479058f-6642-46e3-83a2-2d32949309a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4"
Oct 30 00:37:14.306215 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:14.306186    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0479058f-6642-46e3-83a2-2d32949309a1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5nr4\" (UID: \"0479058f-6642-46e3-83a2-2d32949309a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4"
Oct 30 00:37:14.307244 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:14.307221    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb6kp\" (UniqueName: \"kubernetes.io/projected/0479058f-6642-46e3-83a2-2d32949309a1-kube-api-access-fb6kp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5nr4\" (UID: \"0479058f-6642-46e3-83a2-2d32949309a1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4"
Oct 30 00:37:14.385275 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:14.385242    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4"
Oct 30 00:37:14.721413 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:14.714729    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4"]
Oct 30 00:37:14.726260 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:37:14.726226    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0479058f_6642_46e3_83a2_2d32949309a1.slice/crio-529cc17b3a0a40c5345d0464cc8dd63a9dcdffcfe5e4331b0946f42b8e294292 WatchSource:0}: Error finding container 529cc17b3a0a40c5345d0464cc8dd63a9dcdffcfe5e4331b0946f42b8e294292: Status 404 returned error can't find the container with id 529cc17b3a0a40c5345d0464cc8dd63a9dcdffcfe5e4331b0946f42b8e294292
Oct 30 00:37:14.728313 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:14.728277    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:37:14.728406 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:14.728347    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:37:15.179029 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:15.175744    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4" event={"ID":"0479058f-6642-46e3-83a2-2d32949309a1","Type":"ContainerStarted","Data":"c83927fdc2bb88419d155b81f8aab56fbf1d44e6d48b1074aa7850dce1011914"}
Oct 30 00:37:15.179029 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:15.175849    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4" event={"ID":"0479058f-6642-46e3-83a2-2d32949309a1","Type":"ContainerStarted","Data":"529cc17b3a0a40c5345d0464cc8dd63a9dcdffcfe5e4331b0946f42b8e294292"}
Oct 30 00:37:15.179611 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:15.179347    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4"
Oct 30 00:37:15.180606 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:15.180297    4393 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-k5nr4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 00:37:15.180606 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:15.180352    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4" podUID="0479058f-6642-46e3-83a2-2d32949309a1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 00:37:15.232953 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:15.232892    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4" podStartSLOduration=1.23287765 podStartE2EDuration="1.23287765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:37:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:37:15.232101259 +0000 UTC m=+325.619777655" watchObservedRunningTime="2026-10-30 00:37:15.23287765 +0000 UTC m=+325.620554038"
Oct 30 00:37:16.181345 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:16.181315    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4"
Oct 30 00:37:16.270749 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:16.270716    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g"]
Oct 30 00:37:16.270953 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:16.270933    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g" podUID="ac007403-e5ac-4a40-b5dd-7cc473f6dd49" containerName="discovery" containerID="cri-o://cb807e693f5bde0893e81b3be56a9e497382c3af07fc177f769d9570ce22d822" gracePeriod=30
Oct 30 00:37:16.522194 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:16.522168    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g"
Oct 30 00:37:16.587474 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:16.587447    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-istio-csr-ca-configmap\") pod \"ac007403-e5ac-4a40-b5dd-7cc473f6dd49\" (UID: \"ac007403-e5ac-4a40-b5dd-7cc473f6dd49\") "
Oct 30 00:37:16.587657 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:16.587526    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-istio-token\" (UniqueName: \"kubernetes.io/projected/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-istio-token\") pod \"ac007403-e5ac-4a40-b5dd-7cc473f6dd49\" (UID: \"ac007403-e5ac-4a40-b5dd-7cc473f6dd49\") "
Oct 30 00:37:16.587657 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:16.587556    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-kube-api-access-97smd\" (UniqueName: \"kubernetes.io/projected/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-kube-api-access-97smd\") pod \"ac007403-e5ac-4a40-b5dd-7cc473f6dd49\" (UID: \"ac007403-e5ac-4a40-b5dd-7cc473f6dd49\") "
Oct 30 00:37:16.587657 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:16.587598    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-cacerts\" (UniqueName: \"kubernetes.io/secret/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-cacerts\") pod \"ac007403-e5ac-4a40-b5dd-7cc473f6dd49\" (UID: \"ac007403-e5ac-4a40-b5dd-7cc473f6dd49\") "
Oct 30 00:37:16.587657 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:16.587631    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-istio-csr-dns-cert\") pod \"ac007403-e5ac-4a40-b5dd-7cc473f6dd49\" (UID: \"ac007403-e5ac-4a40-b5dd-7cc473f6dd49\") "
Oct 30 00:37:16.587908 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:16.587658    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-local-certs\") pod \"ac007403-e5ac-4a40-b5dd-7cc473f6dd49\" (UID: \"ac007403-e5ac-4a40-b5dd-7cc473f6dd49\") "
Oct 30 00:37:16.587908 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:16.587729    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-istio-kubeconfig\") pod \"ac007403-e5ac-4a40-b5dd-7cc473f6dd49\" (UID: \"ac007403-e5ac-4a40-b5dd-7cc473f6dd49\") "
Oct 30 00:37:16.588002 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:16.587931    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-istio-csr-ca-configmap" pod "ac007403-e5ac-4a40-b5dd-7cc473f6dd49" (UID: "ac007403-e5ac-4a40-b5dd-7cc473f6dd49"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:37:16.588101 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:16.588083    4393 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-istio-csr-ca-configmap\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:37:16.590313 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:16.590278    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-kube-api-access-97smd" pod "ac007403-e5ac-4a40-b5dd-7cc473f6dd49" (UID: "ac007403-e5ac-4a40-b5dd-7cc473f6dd49"). InnerVolumeSpecName "kube-api-access-97smd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:37:16.590448 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:16.590430    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-istio-token" pod "ac007403-e5ac-4a40-b5dd-7cc473f6dd49" (UID: "ac007403-e5ac-4a40-b5dd-7cc473f6dd49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:37:16.590703 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:16.590677    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-local-certs" pod "ac007403-e5ac-4a40-b5dd-7cc473f6dd49" (UID: "ac007403-e5ac-4a40-b5dd-7cc473f6dd49"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:37:16.590836 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:16.590723    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-istio-csr-dns-cert" pod "ac007403-e5ac-4a40-b5dd-7cc473f6dd49" (UID: "ac007403-e5ac-4a40-b5dd-7cc473f6dd49"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:37:16.590836 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:16.590756    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-cacerts" pod "ac007403-e5ac-4a40-b5dd-7cc473f6dd49" (UID: "ac007403-e5ac-4a40-b5dd-7cc473f6dd49"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:37:16.591096 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:16.591078    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-istio-kubeconfig" pod "ac007403-e5ac-4a40-b5dd-7cc473f6dd49" (UID: "ac007403-e5ac-4a40-b5dd-7cc473f6dd49"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:37:16.689435 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:16.689402    4393 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-istio-csr-dns-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:37:16.689435 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:16.689428    4393 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-local-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:37:16.689435 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:16.689438    4393 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-istio-kubeconfig\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:37:16.689435 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:16.689445    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:37:16.689688 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:16.689453    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97smd\" (UniqueName: \"kubernetes.io/projected/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-kube-api-access-97smd\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:37:16.689688 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:16.689464    4393 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ac007403-e5ac-4a40-b5dd-7cc473f6dd49-cacerts\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:37:17.186223 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:17.186182    4393 generic.go:356] "Generic (PLEG): container finished" podID="ac007403-e5ac-4a40-b5dd-7cc473f6dd49" containerID="cb807e693f5bde0893e81b3be56a9e497382c3af07fc177f769d9570ce22d822" exitCode=0
Oct 30 00:37:17.186675 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:17.186236    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g"
Oct 30 00:37:17.186675 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:17.186263    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g" event={"ID":"ac007403-e5ac-4a40-b5dd-7cc473f6dd49","Type":"ContainerDied","Data":"cb807e693f5bde0893e81b3be56a9e497382c3af07fc177f769d9570ce22d822"}
Oct 30 00:37:17.186675 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:17.186318    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g" event={"ID":"ac007403-e5ac-4a40-b5dd-7cc473f6dd49","Type":"ContainerDied","Data":"1c9fbbe4b984198369ee99cc8787ba037b2dab78454b6666b85b179ec05ea04a"}
Oct 30 00:37:17.186675 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:17.186337    4393 scope.go:122] "RemoveContainer" containerID="cb807e693f5bde0893e81b3be56a9e497382c3af07fc177f769d9570ce22d822"
Oct 30 00:37:17.195286 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:17.195269    4393 scope.go:122] "RemoveContainer" containerID="cb807e693f5bde0893e81b3be56a9e497382c3af07fc177f769d9570ce22d822"
Oct 30 00:37:17.195519 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:37:17.195500    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb807e693f5bde0893e81b3be56a9e497382c3af07fc177f769d9570ce22d822\": container with ID starting with cb807e693f5bde0893e81b3be56a9e497382c3af07fc177f769d9570ce22d822 not found: ID does not exist" containerID="cb807e693f5bde0893e81b3be56a9e497382c3af07fc177f769d9570ce22d822"
Oct 30 00:37:17.195578 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:17.195528    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb807e693f5bde0893e81b3be56a9e497382c3af07fc177f769d9570ce22d822"} err="failed to get container status \"cb807e693f5bde0893e81b3be56a9e497382c3af07fc177f769d9570ce22d822\": rpc error: code = NotFound desc = could not find container \"cb807e693f5bde0893e81b3be56a9e497382c3af07fc177f769d9570ce22d822\": container with ID starting with cb807e693f5bde0893e81b3be56a9e497382c3af07fc177f769d9570ce22d822 not found: ID does not exist"
Oct 30 00:37:17.248040 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:17.248009    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g"]
Oct 30 00:37:17.260177 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:17.260149    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-d695g"]
Oct 30 00:37:17.722808 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:37:17.722765    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac007403-e5ac-4a40-b5dd-7cc473f6dd49" path="/var/lib/kubelet/pods/ac007403-e5ac-4a40-b5dd-7cc473f6dd49/volumes"
Oct 30 00:37:26.509325 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:37:26.509290    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:37:56.509552 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:37:56.509514    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:38:04.353925 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.353885    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdgj7b"]
Oct 30 00:38:04.354433 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.354223    4393 cpu_manager.go:414] "containerMap: removing container" podUID="ac007403-e5ac-4a40-b5dd-7cc473f6dd49" containerName="discovery"
Oct 30 00:38:04.354433 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.354233    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac007403-e5ac-4a40-b5dd-7cc473f6dd49" containerName="discovery"
Oct 30 00:38:04.354433 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.354290    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac007403-e5ac-4a40-b5dd-7cc473f6dd49" containerName="discovery"
Oct 30 00:38:04.357529 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.357509    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdgj7b"
Oct 30 00:38:04.380225 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.380196    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f35c3e21-74d3-43be-b215-f9d58f91c320-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdgj7b\" (UID: \"f35c3e21-74d3-43be-b215-f9d58f91c320\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdgj7b"
Oct 30 00:38:04.380363 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.380263    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f35c3e21-74d3-43be-b215-f9d58f91c320-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdgj7b\" (UID: \"f35c3e21-74d3-43be-b215-f9d58f91c320\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdgj7b"
Oct 30 00:38:04.380363 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.380302    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6k97\" (UniqueName: \"kubernetes.io/projected/f35c3e21-74d3-43be-b215-f9d58f91c320-kube-api-access-c6k97\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdgj7b\" (UID: \"f35c3e21-74d3-43be-b215-f9d58f91c320\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdgj7b"
Oct 30 00:38:04.383395 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.383371    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 00:38:04.383395 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.383389    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-sqqqd\""
Oct 30 00:38:04.383395 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.383378    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 00:38:04.385159 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.385137    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdgj7b"]
Oct 30 00:38:04.441209 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.441184    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap2d7q"]
Oct 30 00:38:04.444672 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.444655    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap2d7q"
Oct 30 00:38:04.458552 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.458531    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-w2pvc\""
Oct 30 00:38:04.464875 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.464847    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap2d7q"]
Oct 30 00:38:04.481392 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.481369    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef6e3fe6-4639-4916-ac90-08c5923e85d5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap2d7q\" (UID: \"ef6e3fe6-4639-4916-ac90-08c5923e85d5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap2d7q"
Oct 30 00:38:04.481480 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.481399    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f35c3e21-74d3-43be-b215-f9d58f91c320-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdgj7b\" (UID: \"f35c3e21-74d3-43be-b215-f9d58f91c320\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdgj7b"
Oct 30 00:38:04.481520 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.481492    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f35c3e21-74d3-43be-b215-f9d58f91c320-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdgj7b\" (UID: \"f35c3e21-74d3-43be-b215-f9d58f91c320\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdgj7b"
Oct 30 00:38:04.481555 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.481538    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6k97\" (UniqueName: \"kubernetes.io/projected/f35c3e21-74d3-43be-b215-f9d58f91c320-kube-api-access-c6k97\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdgj7b\" (UID: \"f35c3e21-74d3-43be-b215-f9d58f91c320\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdgj7b"
Oct 30 00:38:04.481588 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.481568    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef6e3fe6-4639-4916-ac90-08c5923e85d5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap2d7q\" (UID: \"ef6e3fe6-4639-4916-ac90-08c5923e85d5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap2d7q"
Oct 30 00:38:04.481641 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.481624    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjkr5\" (UniqueName: \"kubernetes.io/projected/ef6e3fe6-4639-4916-ac90-08c5923e85d5-kube-api-access-xjkr5\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap2d7q\" (UID: \"ef6e3fe6-4639-4916-ac90-08c5923e85d5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap2d7q"
Oct 30 00:38:04.481853 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.481830    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f35c3e21-74d3-43be-b215-f9d58f91c320-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdgj7b\" (UID: \"f35c3e21-74d3-43be-b215-f9d58f91c320\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdgj7b"
Oct 30 00:38:04.481901 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.481846    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f35c3e21-74d3-43be-b215-f9d58f91c320-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdgj7b\" (UID: \"f35c3e21-74d3-43be-b215-f9d58f91c320\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdgj7b"
Oct 30 00:38:04.504298 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.504264    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6k97\" (UniqueName: \"kubernetes.io/projected/f35c3e21-74d3-43be-b215-f9d58f91c320-kube-api-access-c6k97\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdgj7b\" (UID: \"f35c3e21-74d3-43be-b215-f9d58f91c320\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdgj7b"
Oct 30 00:38:04.537971 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.537942    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7prhv4"]
Oct 30 00:38:04.541625 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.541605    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7prhv4"
Oct 30 00:38:04.555388 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.555369    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-9tlwv\""
Oct 30 00:38:04.560992 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.560970    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7prhv4"]
Oct 30 00:38:04.582366 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.582344    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef6e3fe6-4639-4916-ac90-08c5923e85d5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap2d7q\" (UID: \"ef6e3fe6-4639-4916-ac90-08c5923e85d5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap2d7q"
Oct 30 00:38:04.582472 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.582389    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4a5c2be-5262-4958-ab04-c87fbc4de849-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7prhv4\" (UID: \"e4a5c2be-5262-4958-ab04-c87fbc4de849\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7prhv4"
Oct 30 00:38:04.582472 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.582459    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef6e3fe6-4639-4916-ac90-08c5923e85d5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap2d7q\" (UID: \"ef6e3fe6-4639-4916-ac90-08c5923e85d5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap2d7q"
Oct 30 00:38:04.582581 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.582534    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scqz2\" (UniqueName: \"kubernetes.io/projected/e4a5c2be-5262-4958-ab04-c87fbc4de849-kube-api-access-scqz2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7prhv4\" (UID: \"e4a5c2be-5262-4958-ab04-c87fbc4de849\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7prhv4"
Oct 30 00:38:04.582687 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.582664    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjkr5\" (UniqueName: \"kubernetes.io/projected/ef6e3fe6-4639-4916-ac90-08c5923e85d5-kube-api-access-xjkr5\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap2d7q\" (UID: \"ef6e3fe6-4639-4916-ac90-08c5923e85d5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap2d7q"
Oct 30 00:38:04.582755 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.582704    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4a5c2be-5262-4958-ab04-c87fbc4de849-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7prhv4\" (UID: \"e4a5c2be-5262-4958-ab04-c87fbc4de849\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7prhv4"
Oct 30 00:38:04.582755 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.582663    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef6e3fe6-4639-4916-ac90-08c5923e85d5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap2d7q\" (UID: \"ef6e3fe6-4639-4916-ac90-08c5923e85d5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap2d7q"
Oct 30 00:38:04.582882 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.582867    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef6e3fe6-4639-4916-ac90-08c5923e85d5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap2d7q\" (UID: \"ef6e3fe6-4639-4916-ac90-08c5923e85d5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap2d7q"
Oct 30 00:38:04.605277 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.605213    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjkr5\" (UniqueName: \"kubernetes.io/projected/ef6e3fe6-4639-4916-ac90-08c5923e85d5-kube-api-access-xjkr5\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap2d7q\" (UID: \"ef6e3fe6-4639-4916-ac90-08c5923e85d5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap2d7q"
Oct 30 00:38:04.639382 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.639350    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hgccc"]
Oct 30 00:38:04.643036 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.643018    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hgccc"
Oct 30 00:38:04.657122 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.657105    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-xzzd4\""
Oct 30 00:38:04.662421 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.662398    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hgccc"]
Oct 30 00:38:04.673962 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.673941    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdgj7b"
Oct 30 00:38:04.683030 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.683009    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scqz2\" (UniqueName: \"kubernetes.io/projected/e4a5c2be-5262-4958-ab04-c87fbc4de849-kube-api-access-scqz2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7prhv4\" (UID: \"e4a5c2be-5262-4958-ab04-c87fbc4de849\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7prhv4"
Oct 30 00:38:04.683139 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.683051    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4a5c2be-5262-4958-ab04-c87fbc4de849-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7prhv4\" (UID: \"e4a5c2be-5262-4958-ab04-c87fbc4de849\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7prhv4"
Oct 30 00:38:04.683139 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.683092    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hgccc\" (UID: \"f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hgccc"
Oct 30 00:38:04.683242 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.683141    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4a5c2be-5262-4958-ab04-c87fbc4de849-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7prhv4\" (UID: \"e4a5c2be-5262-4958-ab04-c87fbc4de849\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7prhv4"
Oct 30 00:38:04.683298 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.683215    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hgccc\" (UID: \"f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hgccc"
Oct 30 00:38:04.683298 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.683290    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d2c9\" (UniqueName: \"kubernetes.io/projected/f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c-kube-api-access-6d2c9\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hgccc\" (UID: \"f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hgccc"
Oct 30 00:38:04.683455 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.683394    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4a5c2be-5262-4958-ab04-c87fbc4de849-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7prhv4\" (UID: \"e4a5c2be-5262-4958-ab04-c87fbc4de849\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7prhv4"
Oct 30 00:38:04.683537 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.683486    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4a5c2be-5262-4958-ab04-c87fbc4de849-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7prhv4\" (UID: \"e4a5c2be-5262-4958-ab04-c87fbc4de849\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7prhv4"
Oct 30 00:38:04.706292 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.706259    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scqz2\" (UniqueName: \"kubernetes.io/projected/e4a5c2be-5262-4958-ab04-c87fbc4de849-kube-api-access-scqz2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7prhv4\" (UID: \"e4a5c2be-5262-4958-ab04-c87fbc4de849\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7prhv4"
Oct 30 00:38:04.760122 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.760086    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap2d7q"
Oct 30 00:38:04.784499 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.784472    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d2c9\" (UniqueName: \"kubernetes.io/projected/f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c-kube-api-access-6d2c9\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hgccc\" (UID: \"f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hgccc"
Oct 30 00:38:04.784634 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.784551    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hgccc\" (UID: \"f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hgccc"
Oct 30 00:38:04.784634 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.784598    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hgccc\" (UID: \"f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hgccc"
Oct 30 00:38:04.784971 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.784951    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hgccc\" (UID: \"f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hgccc"
Oct 30 00:38:04.785035 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.784977    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hgccc\" (UID: \"f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hgccc"
Oct 30 00:38:04.808559 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.808536    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d2c9\" (UniqueName: \"kubernetes.io/projected/f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c-kube-api-access-6d2c9\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hgccc\" (UID: \"f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hgccc"
Oct 30 00:38:04.857086 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.857024    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7prhv4"
Oct 30 00:38:04.958703 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.958673    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hgccc"
Oct 30 00:38:04.968554 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:04.968532    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdgj7b"]
Oct 30 00:38:04.970502 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:38:04.970474    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf35c3e21_74d3_43be_b215_f9d58f91c320.slice/crio-1b18a2fa4b3576f23f5e1a4f336c329ce75e1575b8abc1078cbec3fb37f42f80 WatchSource:0}: Error finding container 1b18a2fa4b3576f23f5e1a4f336c329ce75e1575b8abc1078cbec3fb37f42f80: Status 404 returned error can't find the container with id 1b18a2fa4b3576f23f5e1a4f336c329ce75e1575b8abc1078cbec3fb37f42f80
Oct 30 00:38:05.059761 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:05.059740    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap2d7q"]
Oct 30 00:38:05.061499 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:38:05.061468    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef6e3fe6_4639_4916_ac90_08c5923e85d5.slice/crio-a6de99ab556e17580dcf02d92ea2269f21d574e4dce2929725fa26b71fe49df0 WatchSource:0}: Error finding container a6de99ab556e17580dcf02d92ea2269f21d574e4dce2929725fa26b71fe49df0: Status 404 returned error can't find the container with id a6de99ab556e17580dcf02d92ea2269f21d574e4dce2929725fa26b71fe49df0
Oct 30 00:38:05.153325 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:05.153305    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7prhv4"]
Oct 30 00:38:05.155133 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:38:05.155111    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4a5c2be_5262_4958_ab04_c87fbc4de849.slice/crio-e6462ac3569bdc7cd7a6a54940fbcd01bf01fbdaca56eb07a606f0b533f9838c WatchSource:0}: Error finding container e6462ac3569bdc7cd7a6a54940fbcd01bf01fbdaca56eb07a606f0b533f9838c: Status 404 returned error can't find the container with id e6462ac3569bdc7cd7a6a54940fbcd01bf01fbdaca56eb07a606f0b533f9838c
Oct 30 00:38:05.260977 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:05.260954    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hgccc"]
Oct 30 00:38:05.294657 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:38:05.294626    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b33c9e_b74e_48a4_bbf3_c0413f7cec7c.slice/crio-f2aa84c84eb0cb9789bfbbcbccf38f194324a03aeb066f9c57216978ce5e6d90 WatchSource:0}: Error finding container f2aa84c84eb0cb9789bfbbcbccf38f194324a03aeb066f9c57216978ce5e6d90: Status 404 returned error can't find the container with id f2aa84c84eb0cb9789bfbbcbccf38f194324a03aeb066f9c57216978ce5e6d90
Oct 30 00:38:05.395361 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:05.395331    4393 generic.go:356] "Generic (PLEG): container finished" podID="ef6e3fe6-4639-4916-ac90-08c5923e85d5" containerID="d7472ad401542e62c12d47878fd0c5eca67d975245b0e6e74bfb5d2436a4e44a" exitCode=0
Oct 30 00:38:05.395707 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:05.395415    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap2d7q" event={"ID":"ef6e3fe6-4639-4916-ac90-08c5923e85d5","Type":"ContainerDied","Data":"d7472ad401542e62c12d47878fd0c5eca67d975245b0e6e74bfb5d2436a4e44a"}
Oct 30 00:38:05.395707 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:05.395473    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap2d7q" event={"ID":"ef6e3fe6-4639-4916-ac90-08c5923e85d5","Type":"ContainerStarted","Data":"a6de99ab556e17580dcf02d92ea2269f21d574e4dce2929725fa26b71fe49df0"}
Oct 30 00:38:05.396494 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:05.396476    4393 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:38:05.396929 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:05.396909    4393 generic.go:356] "Generic (PLEG): container finished" podID="e4a5c2be-5262-4958-ab04-c87fbc4de849" containerID="c7b15c00ff7c90f3d3b5971d32606ddd8a54e7ff7562d0e83b5dbd9ccbe3032c" exitCode=0
Oct 30 00:38:05.397025 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:05.396999    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7prhv4" event={"ID":"e4a5c2be-5262-4958-ab04-c87fbc4de849","Type":"ContainerDied","Data":"c7b15c00ff7c90f3d3b5971d32606ddd8a54e7ff7562d0e83b5dbd9ccbe3032c"}
Oct 30 00:38:05.397103 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:05.397052    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7prhv4" event={"ID":"e4a5c2be-5262-4958-ab04-c87fbc4de849","Type":"ContainerStarted","Data":"e6462ac3569bdc7cd7a6a54940fbcd01bf01fbdaca56eb07a606f0b533f9838c"}
Oct 30 00:38:05.398792 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:05.398759    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hgccc" event={"ID":"f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c","Type":"ContainerStarted","Data":"571f4dd58e5cb02a53653eb92f2b8685fe106e72f68f25627415b134880ab7dc"}
Oct 30 00:38:05.398871 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:05.398830    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hgccc" event={"ID":"f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c","Type":"ContainerStarted","Data":"f2aa84c84eb0cb9789bfbbcbccf38f194324a03aeb066f9c57216978ce5e6d90"}
Oct 30 00:38:05.400356 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:05.400335    4393 generic.go:356] "Generic (PLEG): container finished" podID="f35c3e21-74d3-43be-b215-f9d58f91c320" containerID="ac4a07d3b52e5fa8afa071aed6fc30c97ca7f6a528af075dc2fa305382b60974" exitCode=0
Oct 30 00:38:05.400402 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:05.400367    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdgj7b" event={"ID":"f35c3e21-74d3-43be-b215-f9d58f91c320","Type":"ContainerDied","Data":"ac4a07d3b52e5fa8afa071aed6fc30c97ca7f6a528af075dc2fa305382b60974"}
Oct 30 00:38:05.400452 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:05.400407    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdgj7b" event={"ID":"f35c3e21-74d3-43be-b215-f9d58f91c320","Type":"ContainerStarted","Data":"1b18a2fa4b3576f23f5e1a4f336c329ce75e1575b8abc1078cbec3fb37f42f80"}
Oct 30 00:38:06.406835 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:06.406774    4393 generic.go:356] "Generic (PLEG): container finished" podID="e4a5c2be-5262-4958-ab04-c87fbc4de849" containerID="e7f3284c67fed407b65d5d21566f3dcd1f66dc5c6a5251b5f223f34c65b1b0e4" exitCode=0
Oct 30 00:38:06.406835 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:06.406816    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7prhv4" event={"ID":"e4a5c2be-5262-4958-ab04-c87fbc4de849","Type":"ContainerDied","Data":"e7f3284c67fed407b65d5d21566f3dcd1f66dc5c6a5251b5f223f34c65b1b0e4"}
Oct 30 00:38:06.408862 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:06.408831    4393 generic.go:356] "Generic (PLEG): container finished" podID="f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c" containerID="571f4dd58e5cb02a53653eb92f2b8685fe106e72f68f25627415b134880ab7dc" exitCode=0
Oct 30 00:38:06.408862 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:06.408859    4393 generic.go:356] "Generic (PLEG): container finished" podID="f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c" containerID="7b13e92f9b1b8e11bd6ef6790e41c10c78da904650d5343a75ee854a59cd30bc" exitCode=0
Oct 30 00:38:06.409003 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:06.408870    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hgccc" event={"ID":"f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c","Type":"ContainerDied","Data":"571f4dd58e5cb02a53653eb92f2b8685fe106e72f68f25627415b134880ab7dc"}
Oct 30 00:38:06.409003 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:06.408923    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hgccc" event={"ID":"f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c","Type":"ContainerDied","Data":"7b13e92f9b1b8e11bd6ef6790e41c10c78da904650d5343a75ee854a59cd30bc"}
Oct 30 00:38:06.410984 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:06.410959    4393 generic.go:356] "Generic (PLEG): container finished" podID="f35c3e21-74d3-43be-b215-f9d58f91c320" containerID="2c8e9d9bd77571304a7dd58f7d6fdb86d91ac91039cb29bf00e94f648d24db0a" exitCode=0
Oct 30 00:38:06.411081 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:06.411041    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdgj7b" event={"ID":"f35c3e21-74d3-43be-b215-f9d58f91c320","Type":"ContainerDied","Data":"2c8e9d9bd77571304a7dd58f7d6fdb86d91ac91039cb29bf00e94f648d24db0a"}
Oct 30 00:38:06.412960 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:06.412936    4393 generic.go:356] "Generic (PLEG): container finished" podID="ef6e3fe6-4639-4916-ac90-08c5923e85d5" containerID="afad5dd73c481437ed5973cbba4c83a055e721a8b6869314eb75e0d5f3aab10d" exitCode=0
Oct 30 00:38:06.413046 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:06.412969    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap2d7q" event={"ID":"ef6e3fe6-4639-4916-ac90-08c5923e85d5","Type":"ContainerDied","Data":"afad5dd73c481437ed5973cbba4c83a055e721a8b6869314eb75e0d5f3aab10d"}
Oct 30 00:38:07.419357 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:07.419320    4393 generic.go:356] "Generic (PLEG): container finished" podID="e4a5c2be-5262-4958-ab04-c87fbc4de849" containerID="ba69596bd22b8b6d588d5d056805a7dc83b21e91179ca9c52269ec5364eb3f1e" exitCode=0
Oct 30 00:38:07.419814 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:07.419407    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7prhv4" event={"ID":"e4a5c2be-5262-4958-ab04-c87fbc4de849","Type":"ContainerDied","Data":"ba69596bd22b8b6d588d5d056805a7dc83b21e91179ca9c52269ec5364eb3f1e"}
Oct 30 00:38:07.421361 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:07.421338    4393 generic.go:356] "Generic (PLEG): container finished" podID="f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c" containerID="01d2bae049eb046a75774247d06fb7c7221d7ad809739893cec252a92e42de20" exitCode=0
Oct 30 00:38:07.421406 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:07.421372    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hgccc" event={"ID":"f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c","Type":"ContainerDied","Data":"01d2bae049eb046a75774247d06fb7c7221d7ad809739893cec252a92e42de20"}
Oct 30 00:38:07.423347 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:07.423326    4393 generic.go:356] "Generic (PLEG): container finished" podID="f35c3e21-74d3-43be-b215-f9d58f91c320" containerID="2cc83e540a8d0a689bddd80cdfbdc67ab7433ae05e1285abc0676f9a51c931f4" exitCode=0
Oct 30 00:38:07.423452 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:07.423374    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdgj7b" event={"ID":"f35c3e21-74d3-43be-b215-f9d58f91c320","Type":"ContainerDied","Data":"2cc83e540a8d0a689bddd80cdfbdc67ab7433ae05e1285abc0676f9a51c931f4"}
Oct 30 00:38:07.425328 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:07.425308    4393 generic.go:356] "Generic (PLEG): container finished" podID="ef6e3fe6-4639-4916-ac90-08c5923e85d5" containerID="4fe97f543d3c29de411fd66a3230e8085b7850430fea2eb47b3aa4f3991f84d1" exitCode=0
Oct 30 00:38:07.425417 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:07.425338    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap2d7q" event={"ID":"ef6e3fe6-4639-4916-ac90-08c5923e85d5","Type":"ContainerDied","Data":"4fe97f543d3c29de411fd66a3230e8085b7850430fea2eb47b3aa4f3991f84d1"}
Oct 30 00:38:08.580558 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.580534    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdgj7b"
Oct 30 00:38:08.618760 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.618738    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f35c3e21-74d3-43be-b215-f9d58f91c320-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f35c3e21-74d3-43be-b215-f9d58f91c320-bundle\") pod \"f35c3e21-74d3-43be-b215-f9d58f91c320\" (UID: \"f35c3e21-74d3-43be-b215-f9d58f91c320\") "
Oct 30 00:38:08.618924 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.618800    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f35c3e21-74d3-43be-b215-f9d58f91c320-kube-api-access-c6k97\" (UniqueName: \"kubernetes.io/projected/f35c3e21-74d3-43be-b215-f9d58f91c320-kube-api-access-c6k97\") pod \"f35c3e21-74d3-43be-b215-f9d58f91c320\" (UID: \"f35c3e21-74d3-43be-b215-f9d58f91c320\") "
Oct 30 00:38:08.618924 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.618833    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f35c3e21-74d3-43be-b215-f9d58f91c320-util\" (UniqueName: \"kubernetes.io/empty-dir/f35c3e21-74d3-43be-b215-f9d58f91c320-util\") pod \"f35c3e21-74d3-43be-b215-f9d58f91c320\" (UID: \"f35c3e21-74d3-43be-b215-f9d58f91c320\") "
Oct 30 00:38:08.619948 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.619840    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f35c3e21-74d3-43be-b215-f9d58f91c320-bundle" pod "f35c3e21-74d3-43be-b215-f9d58f91c320" (UID: "f35c3e21-74d3-43be-b215-f9d58f91c320"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:38:08.622546 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.622203    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f35c3e21-74d3-43be-b215-f9d58f91c320-kube-api-access-c6k97" pod "f35c3e21-74d3-43be-b215-f9d58f91c320" (UID: "f35c3e21-74d3-43be-b215-f9d58f91c320"). InnerVolumeSpecName "kube-api-access-c6k97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:38:08.626709 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.626681    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f35c3e21-74d3-43be-b215-f9d58f91c320-util" pod "f35c3e21-74d3-43be-b215-f9d58f91c320" (UID: "f35c3e21-74d3-43be-b215-f9d58f91c320"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:38:08.627385 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.627360    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap2d7q"
Oct 30 00:38:08.629427 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.629409    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hgccc"
Oct 30 00:38:08.632883 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.632813    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7prhv4"
Oct 30 00:38:08.719600 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.719523    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6e3fe6-4639-4916-ac90-08c5923e85d5-util\" (UniqueName: \"kubernetes.io/empty-dir/ef6e3fe6-4639-4916-ac90-08c5923e85d5-util\") pod \"ef6e3fe6-4639-4916-ac90-08c5923e85d5\" (UID: \"ef6e3fe6-4639-4916-ac90-08c5923e85d5\") "
Oct 30 00:38:08.719600 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.719559    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c-kube-api-access-6d2c9\" (UniqueName: \"kubernetes.io/projected/f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c-kube-api-access-6d2c9\") pod \"f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c\" (UID: \"f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c\") "
Oct 30 00:38:08.719600 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.719597    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c-util\" (UniqueName: \"kubernetes.io/empty-dir/f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c-util\") pod \"f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c\" (UID: \"f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c\") "
Oct 30 00:38:08.719909 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.719620    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef6e3fe6-4639-4916-ac90-08c5923e85d5-kube-api-access-xjkr5\" (UniqueName: \"kubernetes.io/projected/ef6e3fe6-4639-4916-ac90-08c5923e85d5-kube-api-access-xjkr5\") pod \"ef6e3fe6-4639-4916-ac90-08c5923e85d5\" (UID: \"ef6e3fe6-4639-4916-ac90-08c5923e85d5\") "
Oct 30 00:38:08.719909 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.719649    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c-bundle\") pod \"f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c\" (UID: \"f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c\") "
Oct 30 00:38:08.719909 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.719702    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4a5c2be-5262-4958-ab04-c87fbc4de849-kube-api-access-scqz2\" (UniqueName: \"kubernetes.io/projected/e4a5c2be-5262-4958-ab04-c87fbc4de849-kube-api-access-scqz2\") pod \"e4a5c2be-5262-4958-ab04-c87fbc4de849\" (UID: \"e4a5c2be-5262-4958-ab04-c87fbc4de849\") "
Oct 30 00:38:08.719909 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.719728    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6e3fe6-4639-4916-ac90-08c5923e85d5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef6e3fe6-4639-4916-ac90-08c5923e85d5-bundle\") pod \"ef6e3fe6-4639-4916-ac90-08c5923e85d5\" (UID: \"ef6e3fe6-4639-4916-ac90-08c5923e85d5\") "
Oct 30 00:38:08.719909 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.719757    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4a5c2be-5262-4958-ab04-c87fbc4de849-util\" (UniqueName: \"kubernetes.io/empty-dir/e4a5c2be-5262-4958-ab04-c87fbc4de849-util\") pod \"e4a5c2be-5262-4958-ab04-c87fbc4de849\" (UID: \"e4a5c2be-5262-4958-ab04-c87fbc4de849\") "
Oct 30 00:38:08.719909 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.719799    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4a5c2be-5262-4958-ab04-c87fbc4de849-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4a5c2be-5262-4958-ab04-c87fbc4de849-bundle\") pod \"e4a5c2be-5262-4958-ab04-c87fbc4de849\" (UID: \"e4a5c2be-5262-4958-ab04-c87fbc4de849\") "
Oct 30 00:38:08.720210 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.720050    4393 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f35c3e21-74d3-43be-b215-f9d58f91c320-bundle\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:38:08.720210 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.720069    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6k97\" (UniqueName: \"kubernetes.io/projected/f35c3e21-74d3-43be-b215-f9d58f91c320-kube-api-access-c6k97\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:38:08.720210 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.720083    4393 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f35c3e21-74d3-43be-b215-f9d58f91c320-util\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:38:08.720647 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.720620    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c-bundle" pod "f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c" (UID: "f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:38:08.720647 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.720636    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6e3fe6-4639-4916-ac90-08c5923e85d5-bundle" pod "ef6e3fe6-4639-4916-ac90-08c5923e85d5" (UID: "ef6e3fe6-4639-4916-ac90-08c5923e85d5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:38:08.720774 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.720752    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4a5c2be-5262-4958-ab04-c87fbc4de849-bundle" pod "e4a5c2be-5262-4958-ab04-c87fbc4de849" (UID: "e4a5c2be-5262-4958-ab04-c87fbc4de849"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:38:08.722077 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.722055    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c-kube-api-access-6d2c9" pod "f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c" (UID: "f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c"). InnerVolumeSpecName "kube-api-access-6d2c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:38:08.722182 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.722097    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4a5c2be-5262-4958-ab04-c87fbc4de849-kube-api-access-scqz2" pod "e4a5c2be-5262-4958-ab04-c87fbc4de849" (UID: "e4a5c2be-5262-4958-ab04-c87fbc4de849"). InnerVolumeSpecName "kube-api-access-scqz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:38:08.723397 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.723375    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef6e3fe6-4639-4916-ac90-08c5923e85d5-kube-api-access-xjkr5" pod "ef6e3fe6-4639-4916-ac90-08c5923e85d5" (UID: "ef6e3fe6-4639-4916-ac90-08c5923e85d5"). InnerVolumeSpecName "kube-api-access-xjkr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:38:08.728151 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.728125    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4a5c2be-5262-4958-ab04-c87fbc4de849-util" pod "e4a5c2be-5262-4958-ab04-c87fbc4de849" (UID: "e4a5c2be-5262-4958-ab04-c87fbc4de849"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:38:08.728943 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.728899    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6e3fe6-4639-4916-ac90-08c5923e85d5-util" pod "ef6e3fe6-4639-4916-ac90-08c5923e85d5" (UID: "ef6e3fe6-4639-4916-ac90-08c5923e85d5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:38:08.729225 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.729193    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c-util" pod "f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c" (UID: "f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:38:08.820538 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.820494    4393 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c-util\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:38:08.820538 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.820537    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjkr5\" (UniqueName: \"kubernetes.io/projected/ef6e3fe6-4639-4916-ac90-08c5923e85d5-kube-api-access-xjkr5\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:38:08.820538 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.820548    4393 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c-bundle\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:38:08.820760 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.820557    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scqz2\" (UniqueName: \"kubernetes.io/projected/e4a5c2be-5262-4958-ab04-c87fbc4de849-kube-api-access-scqz2\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:38:08.820760 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.820566    4393 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef6e3fe6-4639-4916-ac90-08c5923e85d5-bundle\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:38:08.820760 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.820573    4393 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e4a5c2be-5262-4958-ab04-c87fbc4de849-util\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:38:08.820760 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.820583    4393 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e4a5c2be-5262-4958-ab04-c87fbc4de849-bundle\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:38:08.820760 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.820591    4393 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef6e3fe6-4639-4916-ac90-08c5923e85d5-util\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:38:08.820760 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:08.820599    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d2c9\" (UniqueName: \"kubernetes.io/projected/f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c-kube-api-access-6d2c9\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:38:09.436360 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:09.436329    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap2d7q"
Oct 30 00:38:09.436521 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:09.436356    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap2d7q" event={"ID":"ef6e3fe6-4639-4916-ac90-08c5923e85d5","Type":"ContainerDied","Data":"a6de99ab556e17580dcf02d92ea2269f21d574e4dce2929725fa26b71fe49df0"}
Oct 30 00:38:09.436521 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:09.436436    4393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6de99ab556e17580dcf02d92ea2269f21d574e4dce2929725fa26b71fe49df0"
Oct 30 00:38:09.438485 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:09.438459    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7prhv4" event={"ID":"e4a5c2be-5262-4958-ab04-c87fbc4de849","Type":"ContainerDied","Data":"e6462ac3569bdc7cd7a6a54940fbcd01bf01fbdaca56eb07a606f0b533f9838c"}
Oct 30 00:38:09.438619 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:09.438498    4393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6462ac3569bdc7cd7a6a54940fbcd01bf01fbdaca56eb07a606f0b533f9838c"
Oct 30 00:38:09.438619 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:09.438462    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7prhv4"
Oct 30 00:38:09.440513 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:09.440492    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hgccc"
Oct 30 00:38:09.440513 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:09.440498    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hgccc" event={"ID":"f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c","Type":"ContainerDied","Data":"f2aa84c84eb0cb9789bfbbcbccf38f194324a03aeb066f9c57216978ce5e6d90"}
Oct 30 00:38:09.440649 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:09.440541    4393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2aa84c84eb0cb9789bfbbcbccf38f194324a03aeb066f9c57216978ce5e6d90"
Oct 30 00:38:09.442452 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:09.442426    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdgj7b" event={"ID":"f35c3e21-74d3-43be-b215-f9d58f91c320","Type":"ContainerDied","Data":"1b18a2fa4b3576f23f5e1a4f336c329ce75e1575b8abc1078cbec3fb37f42f80"}
Oct 30 00:38:09.442526 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:09.442463    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdgj7b"
Oct 30 00:38:09.442583 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:09.442468    4393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b18a2fa4b3576f23f5e1a4f336c329ce75e1575b8abc1078cbec3fb37f42f80"
Oct 30 00:38:15.721689 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.721653    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-kvgjz"]
Oct 30 00:38:15.722093 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.721978    4393 cpu_manager.go:414] "containerMap: removing container" podUID="f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c" containerName="util"
Oct 30 00:38:15.722093 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.721990    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c" containerName="util"
Oct 30 00:38:15.722093 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.722001    4393 cpu_manager.go:414] "containerMap: removing container" podUID="f35c3e21-74d3-43be-b215-f9d58f91c320" containerName="pull"
Oct 30 00:38:15.722093 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.722006    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f35c3e21-74d3-43be-b215-f9d58f91c320" containerName="pull"
Oct 30 00:38:15.722093 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.722014    4393 cpu_manager.go:414] "containerMap: removing container" podUID="f35c3e21-74d3-43be-b215-f9d58f91c320" containerName="util"
Oct 30 00:38:15.722093 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.722019    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f35c3e21-74d3-43be-b215-f9d58f91c320" containerName="util"
Oct 30 00:38:15.722093 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.722035    4393 cpu_manager.go:414] "containerMap: removing container" podUID="f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c" containerName="pull"
Oct 30 00:38:15.722093 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.722039    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c" containerName="pull"
Oct 30 00:38:15.722093 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.722045    4393 cpu_manager.go:414] "containerMap: removing container" podUID="ef6e3fe6-4639-4916-ac90-08c5923e85d5" containerName="pull"
Oct 30 00:38:15.722093 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.722049    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef6e3fe6-4639-4916-ac90-08c5923e85d5" containerName="pull"
Oct 30 00:38:15.722093 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.722055    4393 cpu_manager.go:414] "containerMap: removing container" podUID="e4a5c2be-5262-4958-ab04-c87fbc4de849" containerName="pull"
Oct 30 00:38:15.722093 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.722059    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4a5c2be-5262-4958-ab04-c87fbc4de849" containerName="pull"
Oct 30 00:38:15.722093 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.722065    4393 cpu_manager.go:414] "containerMap: removing container" podUID="ef6e3fe6-4639-4916-ac90-08c5923e85d5" containerName="extract"
Oct 30 00:38:15.722093 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.722069    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef6e3fe6-4639-4916-ac90-08c5923e85d5" containerName="extract"
Oct 30 00:38:15.722093 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.722073    4393 cpu_manager.go:414] "containerMap: removing container" podUID="f35c3e21-74d3-43be-b215-f9d58f91c320" containerName="extract"
Oct 30 00:38:15.722093 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.722077    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f35c3e21-74d3-43be-b215-f9d58f91c320" containerName="extract"
Oct 30 00:38:15.722093 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.722087    4393 cpu_manager.go:414] "containerMap: removing container" podUID="f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c" containerName="extract"
Oct 30 00:38:15.722093 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.722091    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c" containerName="extract"
Oct 30 00:38:15.722093 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.722099    4393 cpu_manager.go:414] "containerMap: removing container" podUID="e4a5c2be-5262-4958-ab04-c87fbc4de849" containerName="util"
Oct 30 00:38:15.722093 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.722104    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4a5c2be-5262-4958-ab04-c87fbc4de849" containerName="util"
Oct 30 00:38:15.722692 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.722110    4393 cpu_manager.go:414] "containerMap: removing container" podUID="e4a5c2be-5262-4958-ab04-c87fbc4de849" containerName="extract"
Oct 30 00:38:15.722692 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.722114    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4a5c2be-5262-4958-ab04-c87fbc4de849" containerName="extract"
Oct 30 00:38:15.722692 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.722122    4393 cpu_manager.go:414] "containerMap: removing container" podUID="ef6e3fe6-4639-4916-ac90-08c5923e85d5" containerName="util"
Oct 30 00:38:15.722692 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.722127    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef6e3fe6-4639-4916-ac90-08c5923e85d5" containerName="util"
Oct 30 00:38:15.722692 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.722184    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="f35c3e21-74d3-43be-b215-f9d58f91c320" containerName="extract"
Oct 30 00:38:15.722692 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.722190    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef6e3fe6-4639-4916-ac90-08c5923e85d5" containerName="extract"
Oct 30 00:38:15.722692 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.722196    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b33c9e-b74e-48a4-bbf3-c0413f7cec7c" containerName="extract"
Oct 30 00:38:15.722692 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.722203    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4a5c2be-5262-4958-ab04-c87fbc4de849" containerName="extract"
Oct 30 00:38:15.726475 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.726457    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-kvgjz"
Oct 30 00:38:15.743885 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.743862    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-kvgjz"]
Oct 30 00:38:15.753060 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.753038    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 00:38:15.753215 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.753081    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 00:38:15.753215 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.753169    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-72bqv\""
Oct 30 00:38:15.782166 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.782136    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trrsm\" (UniqueName: \"kubernetes.io/projected/7bd77191-8d97-498a-978b-0765096177a1-kube-api-access-trrsm\") pod \"authorino-operator-68d6dd76dc-kvgjz\" (UID: \"7bd77191-8d97-498a-978b-0765096177a1\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-kvgjz"
Oct 30 00:38:15.883328 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.883291    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trrsm\" (UniqueName: \"kubernetes.io/projected/7bd77191-8d97-498a-978b-0765096177a1-kube-api-access-trrsm\") pod \"authorino-operator-68d6dd76dc-kvgjz\" (UID: \"7bd77191-8d97-498a-978b-0765096177a1\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-kvgjz"
Oct 30 00:38:15.906289 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:15.906258    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trrsm\" (UniqueName: \"kubernetes.io/projected/7bd77191-8d97-498a-978b-0765096177a1-kube-api-access-trrsm\") pod \"authorino-operator-68d6dd76dc-kvgjz\" (UID: \"7bd77191-8d97-498a-978b-0765096177a1\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-kvgjz"
Oct 30 00:38:16.038705 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:16.038611    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-kvgjz"
Oct 30 00:38:16.335965 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:16.335939    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-kvgjz"]
Oct 30 00:38:16.337754 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:38:16.337730    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd77191_8d97_498a_978b_0765096177a1.slice/crio-d7966258a2ac046c14c7962e1bde5483f1e13827869ea4bb7b53fcbf80135fe4 WatchSource:0}: Error finding container d7966258a2ac046c14c7962e1bde5483f1e13827869ea4bb7b53fcbf80135fe4: Status 404 returned error can't find the container with id d7966258a2ac046c14c7962e1bde5483f1e13827869ea4bb7b53fcbf80135fe4
Oct 30 00:38:16.475652 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:16.475620    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-kvgjz" event={"ID":"7bd77191-8d97-498a-978b-0765096177a1","Type":"ContainerStarted","Data":"d7966258a2ac046c14c7962e1bde5483f1e13827869ea4bb7b53fcbf80135fe4"}
Oct 30 00:38:17.906117 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:17.906079    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-xgjkr"]
Oct 30 00:38:17.914963 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:17.914566    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-xgjkr"
Oct 30 00:38:17.933450 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:17.933428    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 30 00:38:17.933583 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:17.933552    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-vq9b2\""
Oct 30 00:38:17.947253 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:17.947231    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-xgjkr"]
Oct 30 00:38:18.003439 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:18.003409    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjpj5\" (UniqueName: \"kubernetes.io/projected/4c9fdd3e-ff82-404c-a637-1b18a139938d-kube-api-access-hjpj5\") pod \"dns-operator-controller-manager-65bfd79598-xgjkr\" (UID: \"4c9fdd3e-ff82-404c-a637-1b18a139938d\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-xgjkr"
Oct 30 00:38:18.104901 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:18.104862    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjpj5\" (UniqueName: \"kubernetes.io/projected/4c9fdd3e-ff82-404c-a637-1b18a139938d-kube-api-access-hjpj5\") pod \"dns-operator-controller-manager-65bfd79598-xgjkr\" (UID: \"4c9fdd3e-ff82-404c-a637-1b18a139938d\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-xgjkr"
Oct 30 00:38:18.137012 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:18.136985    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjpj5\" (UniqueName: \"kubernetes.io/projected/4c9fdd3e-ff82-404c-a637-1b18a139938d-kube-api-access-hjpj5\") pod \"dns-operator-controller-manager-65bfd79598-xgjkr\" (UID: \"4c9fdd3e-ff82-404c-a637-1b18a139938d\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-xgjkr"
Oct 30 00:38:18.230153 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:18.230082    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-xgjkr"
Oct 30 00:38:18.551972 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:18.551943    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-xgjkr"]
Oct 30 00:38:18.554134 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:38:18.554104    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c9fdd3e_ff82_404c_a637_1b18a139938d.slice/crio-fc18c2b6efeb6fe74bae81b57a32978cb29e1d63bf48025a7c5657daf63e95bb WatchSource:0}: Error finding container fc18c2b6efeb6fe74bae81b57a32978cb29e1d63bf48025a7c5657daf63e95bb: Status 404 returned error can't find the container with id fc18c2b6efeb6fe74bae81b57a32978cb29e1d63bf48025a7c5657daf63e95bb
Oct 30 00:38:19.498339 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:19.498293    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-kvgjz" event={"ID":"7bd77191-8d97-498a-978b-0765096177a1","Type":"ContainerStarted","Data":"7e9412fd435c831bde2a301e22f504be627a1c460839793c3b5fb1545700dcae"}
Oct 30 00:38:19.498832 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:19.498498    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-kvgjz"
Oct 30 00:38:19.499585 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:19.499566    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-xgjkr" event={"ID":"4c9fdd3e-ff82-404c-a637-1b18a139938d","Type":"ContainerStarted","Data":"fc18c2b6efeb6fe74bae81b57a32978cb29e1d63bf48025a7c5657daf63e95bb"}
Oct 30 00:38:19.543543 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:19.543495    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d6dd76dc-kvgjz" podStartSLOduration=2.129567394 podStartE2EDuration="4.543480497s" totalImagesPullingTime="2.413913103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:19.541973357 +0000 UTC m=+389.929649747" watchObservedRunningTime="2026-10-30 00:38:19.543480497 +0000 UTC m=+389.931156886"
Oct 30 00:38:20.506361 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:20.506323    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-xgjkr" event={"ID":"4c9fdd3e-ff82-404c-a637-1b18a139938d","Type":"ContainerStarted","Data":"6fc76cce9029bbded5c20168d6a6ddee49b960c9317a0e639ec9e82a0c5fad93"}
Oct 30 00:38:20.506799 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:20.506565    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-xgjkr"
Oct 30 00:38:20.550258 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:20.550206    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-xgjkr" podStartSLOduration=2.025468938 podStartE2EDuration="3.550194278s" totalImagesPullingTime="1.52472534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:20.548867204 +0000 UTC m=+390.936543593" watchObservedRunningTime="2026-10-30 00:38:20.550194278 +0000 UTC m=+390.937870666"
Oct 30 00:38:22.218061 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:22.218032    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xns27"]
Oct 30 00:38:22.221507 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:22.221490    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xns27"
Oct 30 00:38:22.238524 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:22.238504    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xns27"]
Oct 30 00:38:22.248027 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:22.248002    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-pj9zs\""
Oct 30 00:38:22.346920 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:22.346897    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-xns27\" (UID: \"ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xns27"
Oct 30 00:38:22.347034 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:22.346930    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wlz9\" (UniqueName: \"kubernetes.io/projected/ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5-kube-api-access-4wlz9\") pod \"kuadrant-operator-controller-manager-6b6fb56846-xns27\" (UID: \"ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xns27"
Oct 30 00:38:22.448220 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:22.448192    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-xns27\" (UID: \"ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xns27"
Oct 30 00:38:22.448333 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:22.448227    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wlz9\" (UniqueName: \"kubernetes.io/projected/ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5-kube-api-access-4wlz9\") pod \"kuadrant-operator-controller-manager-6b6fb56846-xns27\" (UID: \"ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xns27"
Oct 30 00:38:22.448563 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:22.448544    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-xns27\" (UID: \"ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xns27"
Oct 30 00:38:22.471554 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:22.471490    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wlz9\" (UniqueName: \"kubernetes.io/projected/ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5-kube-api-access-4wlz9\") pod \"kuadrant-operator-controller-manager-6b6fb56846-xns27\" (UID: \"ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xns27"
Oct 30 00:38:22.534628 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:22.534603    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xns27"
Oct 30 00:38:22.870702 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:22.870667    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xns27"]
Oct 30 00:38:22.872822 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:38:22.872797    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce3ba8e1_75df_4d00_a91a_7d366b4ca5f5.slice/crio-8b96eb6535ea98b38e4157c227adf261324ca9ddeabbf38d727d2afc7ca4727e WatchSource:0}: Error finding container 8b96eb6535ea98b38e4157c227adf261324ca9ddeabbf38d727d2afc7ca4727e: Status 404 returned error can't find the container with id 8b96eb6535ea98b38e4157c227adf261324ca9ddeabbf38d727d2afc7ca4727e
Oct 30 00:38:23.523898 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:23.523863    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xns27" event={"ID":"ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5","Type":"ContainerStarted","Data":"8b96eb6535ea98b38e4157c227adf261324ca9ddeabbf38d727d2afc7ca4727e"}
Oct 30 00:38:26.519032 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:38:26.518988    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:38:28.553073 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:28.553034    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xns27" event={"ID":"ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5","Type":"ContainerStarted","Data":"4f504ce3077cc526c2c9943994e4058b5cff77852cebe54de4f2447f78662194"}
Oct 30 00:38:28.553538 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:28.553152    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xns27"
Oct 30 00:38:28.618680 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:28.618619    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xns27" podStartSLOduration=1.811215279 podStartE2EDuration="6.618605271s" totalImagesPullingTime="4.807389992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:28.61762584 +0000 UTC m=+399.005302229" watchObservedRunningTime="2026-10-30 00:38:28.618605271 +0000 UTC m=+399.006281661"
Oct 30 00:38:30.509062 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:30.509032    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-kvgjz"
Oct 30 00:38:31.514591 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:31.514564    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-xgjkr"
Oct 30 00:38:39.560890 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:39.560859    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xns27"
Oct 30 00:38:40.908178 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:40.908138    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-nb4dc"]
Oct 30 00:38:40.913178 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:40.913154    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-nb4dc"
Oct 30 00:38:40.929521 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:40.929489    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-nb4dc"]
Oct 30 00:38:41.014025 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:41.013979    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a4052cf-2898-418f-b4a1-ae6a1da1989c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-nb4dc\" (UID: \"4a4052cf-2898-418f-b4a1-ae6a1da1989c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-nb4dc"
Oct 30 00:38:41.014222 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:41.014114    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdkpr\" (UniqueName: \"kubernetes.io/projected/4a4052cf-2898-418f-b4a1-ae6a1da1989c-kube-api-access-vdkpr\") pod \"kuadrant-operator-controller-manager-6b76784d48-nb4dc\" (UID: \"4a4052cf-2898-418f-b4a1-ae6a1da1989c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-nb4dc"
Oct 30 00:38:41.115245 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:41.115208    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdkpr\" (UniqueName: \"kubernetes.io/projected/4a4052cf-2898-418f-b4a1-ae6a1da1989c-kube-api-access-vdkpr\") pod \"kuadrant-operator-controller-manager-6b76784d48-nb4dc\" (UID: \"4a4052cf-2898-418f-b4a1-ae6a1da1989c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-nb4dc"
Oct 30 00:38:41.115438 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:41.115259    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a4052cf-2898-418f-b4a1-ae6a1da1989c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-nb4dc\" (UID: \"4a4052cf-2898-418f-b4a1-ae6a1da1989c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-nb4dc"
Oct 30 00:38:41.115648 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:41.115630    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a4052cf-2898-418f-b4a1-ae6a1da1989c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-nb4dc\" (UID: \"4a4052cf-2898-418f-b4a1-ae6a1da1989c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-nb4dc"
Oct 30 00:38:41.138161 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:41.138132    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdkpr\" (UniqueName: \"kubernetes.io/projected/4a4052cf-2898-418f-b4a1-ae6a1da1989c-kube-api-access-vdkpr\") pod \"kuadrant-operator-controller-manager-6b76784d48-nb4dc\" (UID: \"4a4052cf-2898-418f-b4a1-ae6a1da1989c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-nb4dc"
Oct 30 00:38:41.226204 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:41.226107    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-nb4dc"
Oct 30 00:38:41.560485 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:41.560463    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-nb4dc"]
Oct 30 00:38:41.562121 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:38:41.562088    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a4052cf_2898_418f_b4a1_ae6a1da1989c.slice/crio-cbdb21fde24004229d18e2223e7929909e609d8e981afa86eb9eb2c0d2d2c27b WatchSource:0}: Error finding container cbdb21fde24004229d18e2223e7929909e609d8e981afa86eb9eb2c0d2d2c27b: Status 404 returned error can't find the container with id cbdb21fde24004229d18e2223e7929909e609d8e981afa86eb9eb2c0d2d2c27b
Oct 30 00:38:41.615331 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:41.615307    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-nb4dc" event={"ID":"4a4052cf-2898-418f-b4a1-ae6a1da1989c","Type":"ContainerStarted","Data":"cbdb21fde24004229d18e2223e7929909e609d8e981afa86eb9eb2c0d2d2c27b"}
Oct 30 00:38:42.621985 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:42.621945    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-nb4dc" event={"ID":"4a4052cf-2898-418f-b4a1-ae6a1da1989c","Type":"ContainerStarted","Data":"846c346d92421854ac2fbb431ab30feb9557eb658aae6b45ba13f4702252c864"}
Oct 30 00:38:42.622428 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:42.622074    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-nb4dc"
Oct 30 00:38:42.689419 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:42.689368    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-nb4dc" podStartSLOduration=2.689353872 podStartE2EDuration="2.689353872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:42.676185019 +0000 UTC m=+413.063861419" watchObservedRunningTime="2026-10-30 00:38:42.689353872 +0000 UTC m=+413.077030261"
Oct 30 00:38:43.236335 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.236302    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xns27"]
Oct 30 00:38:43.236532 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.236492    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xns27" podUID="ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5" containerName="manager" containerID="cri-o://4f504ce3077cc526c2c9943994e4058b5cff77852cebe54de4f2447f78662194" gracePeriod=2
Oct 30 00:38:43.249176 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.249147    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xns27"]
Oct 30 00:38:43.251160 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:38:43.251124    4393 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-xns27\" is forbidden: User \"system:node:ip-10-0-135-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-56.ec2.internal' and this object" podUID="ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xns27"
Oct 30 00:38:43.266488 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.266464    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wgfbs"]
Oct 30 00:38:43.267010 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.266991    4393 cpu_manager.go:414] "containerMap: removing container" podUID="ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5" containerName="manager"
Oct 30 00:38:43.267100 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.267012    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5" containerName="manager"
Oct 30 00:38:43.267163 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.267098    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5" containerName="manager"
Oct 30 00:38:43.270096 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.270077    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-nb4dc"]
Oct 30 00:38:43.270203 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.270190    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wgfbs"
Oct 30 00:38:43.281588 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.281560    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-nb4dc"]
Oct 30 00:38:43.297099 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.297055    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wgfbs"]
Oct 30 00:38:43.317002 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.316969    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-sppzp"]
Oct 30 00:38:43.317436 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.317413    4393 cpu_manager.go:414] "containerMap: removing container" podUID="4a4052cf-2898-418f-b4a1-ae6a1da1989c" containerName="manager"
Oct 30 00:38:43.317436 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.317436    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a4052cf-2898-418f-b4a1-ae6a1da1989c" containerName="manager"
Oct 30 00:38:43.317622 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.317528    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a4052cf-2898-418f-b4a1-ae6a1da1989c" containerName="manager"
Oct 30 00:38:43.321426 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.320819    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-sppzp"
Oct 30 00:38:43.333150 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:38:43.333112    4393 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-xns27\" is forbidden: User \"system:node:ip-10-0-135-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-56.ec2.internal' and this object" podUID="ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xns27"
Oct 30 00:38:43.341803 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.341759    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-sppzp\" (UID: \"0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-sppzp"
Oct 30 00:38:43.341954 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.341910    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ngt8\" (UniqueName: \"kubernetes.io/projected/31dbc008-f873-4d22-83d8-aaf31d2a9cb8-kube-api-access-6ngt8\") pod \"kuadrant-operator-controller-manager-6b6fb56846-wgfbs\" (UID: \"31dbc008-f873-4d22-83d8-aaf31d2a9cb8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wgfbs"
Oct 30 00:38:43.342043 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.342023    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/31dbc008-f873-4d22-83d8-aaf31d2a9cb8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-wgfbs\" (UID: \"31dbc008-f873-4d22-83d8-aaf31d2a9cb8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wgfbs"
Oct 30 00:38:43.342114 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.342072    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86hlj\" (UniqueName: \"kubernetes.io/projected/0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43-kube-api-access-86hlj\") pod \"kuadrant-operator-controller-manager-6b76784d48-sppzp\" (UID: \"0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-sppzp"
Oct 30 00:38:43.349243 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:38:43.349210    4393 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-xns27\" is forbidden: User \"system:node:ip-10-0-135-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-56.ec2.internal' and this object" podUID="ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xns27"
Oct 30 00:38:43.353882 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.353855    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-sppzp"]
Oct 30 00:38:43.442457 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.442422    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/31dbc008-f873-4d22-83d8-aaf31d2a9cb8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-wgfbs\" (UID: \"31dbc008-f873-4d22-83d8-aaf31d2a9cb8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wgfbs"
Oct 30 00:38:43.442633 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.442465    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86hlj\" (UniqueName: \"kubernetes.io/projected/0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43-kube-api-access-86hlj\") pod \"kuadrant-operator-controller-manager-6b76784d48-sppzp\" (UID: \"0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-sppzp"
Oct 30 00:38:43.442633 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.442496    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-sppzp\" (UID: \"0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-sppzp"
Oct 30 00:38:43.442633 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.442545    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ngt8\" (UniqueName: \"kubernetes.io/projected/31dbc008-f873-4d22-83d8-aaf31d2a9cb8-kube-api-access-6ngt8\") pod \"kuadrant-operator-controller-manager-6b6fb56846-wgfbs\" (UID: \"31dbc008-f873-4d22-83d8-aaf31d2a9cb8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wgfbs"
Oct 30 00:38:43.442941 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.442916    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-sppzp\" (UID: \"0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-sppzp"
Oct 30 00:38:43.443023 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.442937    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/31dbc008-f873-4d22-83d8-aaf31d2a9cb8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-wgfbs\" (UID: \"31dbc008-f873-4d22-83d8-aaf31d2a9cb8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wgfbs"
Oct 30 00:38:43.466209 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.466182    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ngt8\" (UniqueName: \"kubernetes.io/projected/31dbc008-f873-4d22-83d8-aaf31d2a9cb8-kube-api-access-6ngt8\") pod \"kuadrant-operator-controller-manager-6b6fb56846-wgfbs\" (UID: \"31dbc008-f873-4d22-83d8-aaf31d2a9cb8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wgfbs"
Oct 30 00:38:43.466315 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.466266    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86hlj\" (UniqueName: \"kubernetes.io/projected/0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43-kube-api-access-86hlj\") pod \"kuadrant-operator-controller-manager-6b76784d48-sppzp\" (UID: \"0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-sppzp"
Oct 30 00:38:43.488361 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.488311    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xns27"
Oct 30 00:38:43.501265 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:38:43.501238    4393 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-xns27\" is forbidden: User \"system:node:ip-10-0-135-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-56.ec2.internal' and this object" podUID="ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xns27"
Oct 30 00:38:43.543622 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.543595    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5-kube-api-access-4wlz9\" (UniqueName: \"kubernetes.io/projected/ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5-kube-api-access-4wlz9\") pod \"ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5\" (UID: \"ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5\") "
Oct 30 00:38:43.543739 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.543657    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5-extensions-socket-volume\") pod \"ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5\" (UID: \"ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5\") "
Oct 30 00:38:43.544134 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.544115    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5-extensions-socket-volume" pod "ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5" (UID: "ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:38:43.545942 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.545922    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5-kube-api-access-4wlz9" pod "ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5" (UID: "ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5"). InnerVolumeSpecName "kube-api-access-4wlz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:38:43.627808 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.627761    4393 generic.go:356] "Generic (PLEG): container finished" podID="ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5" containerID="4f504ce3077cc526c2c9943994e4058b5cff77852cebe54de4f2447f78662194" exitCode=0
Oct 30 00:38:43.628224 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.627834    4393 scope.go:122] "RemoveContainer" containerID="4f504ce3077cc526c2c9943994e4058b5cff77852cebe54de4f2447f78662194"
Oct 30 00:38:43.628224 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.627836    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xns27"
Oct 30 00:38:43.628387 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.628372    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wgfbs"
Oct 30 00:38:43.638857 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.638839    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-sppzp"
Oct 30 00:38:43.641329 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:38:43.641301    4393 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-xns27\" is forbidden: User \"system:node:ip-10-0-135-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-56.ec2.internal' and this object" podUID="ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xns27"
Oct 30 00:38:43.642218 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.642194    4393 scope.go:122] "RemoveContainer" containerID="4f504ce3077cc526c2c9943994e4058b5cff77852cebe54de4f2447f78662194"
Oct 30 00:38:43.642667 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:38:43.642642    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f504ce3077cc526c2c9943994e4058b5cff77852cebe54de4f2447f78662194\": container with ID starting with 4f504ce3077cc526c2c9943994e4058b5cff77852cebe54de4f2447f78662194 not found: ID does not exist" containerID="4f504ce3077cc526c2c9943994e4058b5cff77852cebe54de4f2447f78662194"
Oct 30 00:38:43.642734 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.642688    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f504ce3077cc526c2c9943994e4058b5cff77852cebe54de4f2447f78662194"} err="failed to get container status \"4f504ce3077cc526c2c9943994e4058b5cff77852cebe54de4f2447f78662194\": rpc error: code = NotFound desc = could not find container \"4f504ce3077cc526c2c9943994e4058b5cff77852cebe54de4f2447f78662194\": container with ID starting with 4f504ce3077cc526c2c9943994e4058b5cff77852cebe54de4f2447f78662194 not found: ID does not exist"
Oct 30 00:38:43.645220 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.645193    4393 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5-extensions-socket-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:38:43.645319 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.645226    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wlz9\" (UniqueName: \"kubernetes.io/projected/ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5-kube-api-access-4wlz9\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:38:43.654385 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:38:43.654360    4393 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-xns27\" is forbidden: User \"system:node:ip-10-0-135-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-56.ec2.internal' and this object" podUID="ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-xns27"
Oct 30 00:38:43.723063 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.723038    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5" path="/var/lib/kubelet/pods/ce3ba8e1-75df-4d00-a91a-7d366b4ca5f5/volumes"
Oct 30 00:38:43.948386 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.948360    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wgfbs"]
Oct 30 00:38:43.949993 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:38:43.949971    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31dbc008_f873_4d22_83d8_aaf31d2a9cb8.slice/crio-29ddcb548262cec435bbf5ff6c93c2011e171b704c04c318109e5c59c303b2e8 WatchSource:0}: Error finding container 29ddcb548262cec435bbf5ff6c93c2011e171b704c04c318109e5c59c303b2e8: Status 404 returned error can't find the container with id 29ddcb548262cec435bbf5ff6c93c2011e171b704c04c318109e5c59c303b2e8
Oct 30 00:38:43.960114 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:43.960031    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-sppzp"]
Oct 30 00:38:43.963454 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:38:43.963430    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd5ac6a_e3d7_41e2_aaf8_57a149ecfd43.slice/crio-7cbfcb012e93a0a2f552fa6984e40c55dc781ca21d459f4665510d299bf6e36d WatchSource:0}: Error finding container 7cbfcb012e93a0a2f552fa6984e40c55dc781ca21d459f4665510d299bf6e36d: Status 404 returned error can't find the container with id 7cbfcb012e93a0a2f552fa6984e40c55dc781ca21d459f4665510d299bf6e36d
Oct 30 00:38:44.634386 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:44.634350    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wgfbs" event={"ID":"31dbc008-f873-4d22-83d8-aaf31d2a9cb8","Type":"ContainerStarted","Data":"7f7aaf45321a74eee8d5a4f7469c4f1c815a9483f04679403d02d0938f6d8907"}
Oct 30 00:38:44.634906 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:44.634409    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wgfbs" event={"ID":"31dbc008-f873-4d22-83d8-aaf31d2a9cb8","Type":"ContainerStarted","Data":"29ddcb548262cec435bbf5ff6c93c2011e171b704c04c318109e5c59c303b2e8"}
Oct 30 00:38:44.634906 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:44.634438    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wgfbs"
Oct 30 00:38:44.637065 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:44.637039    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-sppzp" event={"ID":"0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43","Type":"ContainerStarted","Data":"072fe61d9937e495eec008a239aa500f1343807e6898add130d24276001988b8"}
Oct 30 00:38:44.637176 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:44.637088    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-sppzp"
Oct 30 00:38:44.637176 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:44.637110    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-sppzp" event={"ID":"0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43","Type":"ContainerStarted","Data":"7cbfcb012e93a0a2f552fa6984e40c55dc781ca21d459f4665510d299bf6e36d"}
Oct 30 00:38:44.637176 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:44.637158    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-nb4dc" podUID="4a4052cf-2898-418f-b4a1-ae6a1da1989c" containerName="manager" containerID="cri-o://846c346d92421854ac2fbb431ab30feb9557eb658aae6b45ba13f4702252c864" gracePeriod=2
Oct 30 00:38:44.706402 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:44.706347    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wgfbs" podStartSLOduration=1.706331449 podStartE2EDuration="1.706331449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:44.692850737 +0000 UTC m=+415.080527127" watchObservedRunningTime="2026-10-30 00:38:44.706331449 +0000 UTC m=+415.094007832"
Oct 30 00:38:44.744719 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:44.744662    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-sppzp" podStartSLOduration=1.744648138 podStartE2EDuration="1.744648138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:38:44.742873699 +0000 UTC m=+415.130550079" watchObservedRunningTime="2026-10-30 00:38:44.744648138 +0000 UTC m=+415.132324527"
Oct 30 00:38:44.900226 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:44.900203    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-nb4dc"
Oct 30 00:38:44.956482 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:44.956443    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a4052cf-2898-418f-b4a1-ae6a1da1989c-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a4052cf-2898-418f-b4a1-ae6a1da1989c-extensions-socket-volume\") pod \"4a4052cf-2898-418f-b4a1-ae6a1da1989c\" (UID: \"4a4052cf-2898-418f-b4a1-ae6a1da1989c\") "
Oct 30 00:38:44.956649 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:44.956598    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a4052cf-2898-418f-b4a1-ae6a1da1989c-kube-api-access-vdkpr\" (UniqueName: \"kubernetes.io/projected/4a4052cf-2898-418f-b4a1-ae6a1da1989c-kube-api-access-vdkpr\") pod \"4a4052cf-2898-418f-b4a1-ae6a1da1989c\" (UID: \"4a4052cf-2898-418f-b4a1-ae6a1da1989c\") "
Oct 30 00:38:44.956941 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:44.956918    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a4052cf-2898-418f-b4a1-ae6a1da1989c-extensions-socket-volume" pod "4a4052cf-2898-418f-b4a1-ae6a1da1989c" (UID: "4a4052cf-2898-418f-b4a1-ae6a1da1989c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:38:44.958811 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:44.958773    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a4052cf-2898-418f-b4a1-ae6a1da1989c-kube-api-access-vdkpr" pod "4a4052cf-2898-418f-b4a1-ae6a1da1989c" (UID: "4a4052cf-2898-418f-b4a1-ae6a1da1989c"). InnerVolumeSpecName "kube-api-access-vdkpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:38:45.057307 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:45.057272    4393 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a4052cf-2898-418f-b4a1-ae6a1da1989c-extensions-socket-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:38:45.057307 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:45.057302    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdkpr\" (UniqueName: \"kubernetes.io/projected/4a4052cf-2898-418f-b4a1-ae6a1da1989c-kube-api-access-vdkpr\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:38:45.644168 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:45.644131    4393 generic.go:356] "Generic (PLEG): container finished" podID="4a4052cf-2898-418f-b4a1-ae6a1da1989c" containerID="846c346d92421854ac2fbb431ab30feb9557eb658aae6b45ba13f4702252c864" exitCode=0
Oct 30 00:38:45.644665 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:45.644186    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-nb4dc"
Oct 30 00:38:45.644665 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:45.644226    4393 scope.go:122] "RemoveContainer" containerID="846c346d92421854ac2fbb431ab30feb9557eb658aae6b45ba13f4702252c864"
Oct 30 00:38:45.653371 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:45.653352    4393 scope.go:122] "RemoveContainer" containerID="846c346d92421854ac2fbb431ab30feb9557eb658aae6b45ba13f4702252c864"
Oct 30 00:38:45.653631 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:38:45.653612    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"846c346d92421854ac2fbb431ab30feb9557eb658aae6b45ba13f4702252c864\": container with ID starting with 846c346d92421854ac2fbb431ab30feb9557eb658aae6b45ba13f4702252c864 not found: ID does not exist" containerID="846c346d92421854ac2fbb431ab30feb9557eb658aae6b45ba13f4702252c864"
Oct 30 00:38:45.653675 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:45.653641    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"846c346d92421854ac2fbb431ab30feb9557eb658aae6b45ba13f4702252c864"} err="failed to get container status \"846c346d92421854ac2fbb431ab30feb9557eb658aae6b45ba13f4702252c864\": rpc error: code = NotFound desc = could not find container \"846c346d92421854ac2fbb431ab30feb9557eb658aae6b45ba13f4702252c864\": container with ID starting with 846c346d92421854ac2fbb431ab30feb9557eb658aae6b45ba13f4702252c864 not found: ID does not exist"
Oct 30 00:38:45.657380 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:38:45.657357    4393 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-nb4dc\" is forbidden: User \"system:node:ip-10-0-135-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-56.ec2.internal' and this object" podUID="4a4052cf-2898-418f-b4a1-ae6a1da1989c" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-nb4dc"
Oct 30 00:38:45.670234 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:38:45.670211    4393 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-nb4dc\" is forbidden: User \"system:node:ip-10-0-135-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-56.ec2.internal' and this object" podUID="4a4052cf-2898-418f-b4a1-ae6a1da1989c" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-nb4dc"
Oct 30 00:38:45.727520 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:45.727481    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a4052cf-2898-418f-b4a1-ae6a1da1989c" path="/var/lib/kubelet/pods/4a4052cf-2898-418f-b4a1-ae6a1da1989c/volumes"
Oct 30 00:38:55.646349 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:55.646317    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wgfbs"
Oct 30 00:38:55.646855 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:55.646404    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-sppzp"
Oct 30 00:38:55.738171 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:55.738143    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wgfbs"]
Oct 30 00:38:55.738364 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:55.738337    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wgfbs" podUID="31dbc008-f873-4d22-83d8-aaf31d2a9cb8" containerName="manager" containerID="cri-o://7f7aaf45321a74eee8d5a4f7469c4f1c815a9483f04679403d02d0938f6d8907" gracePeriod=10
Oct 30 00:38:55.987278 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:55.987243    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wgfbs"
Oct 30 00:38:56.051321 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:56.051293    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31dbc008-f873-4d22-83d8-aaf31d2a9cb8-kube-api-access-6ngt8\" (UniqueName: \"kubernetes.io/projected/31dbc008-f873-4d22-83d8-aaf31d2a9cb8-kube-api-access-6ngt8\") pod \"31dbc008-f873-4d22-83d8-aaf31d2a9cb8\" (UID: \"31dbc008-f873-4d22-83d8-aaf31d2a9cb8\") "
Oct 30 00:38:56.051469 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:56.051405    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31dbc008-f873-4d22-83d8-aaf31d2a9cb8-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/31dbc008-f873-4d22-83d8-aaf31d2a9cb8-extensions-socket-volume\") pod \"31dbc008-f873-4d22-83d8-aaf31d2a9cb8\" (UID: \"31dbc008-f873-4d22-83d8-aaf31d2a9cb8\") "
Oct 30 00:38:56.051746 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:56.051726    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31dbc008-f873-4d22-83d8-aaf31d2a9cb8-extensions-socket-volume" pod "31dbc008-f873-4d22-83d8-aaf31d2a9cb8" (UID: "31dbc008-f873-4d22-83d8-aaf31d2a9cb8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:38:56.053460 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:56.053437    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31dbc008-f873-4d22-83d8-aaf31d2a9cb8-kube-api-access-6ngt8" pod "31dbc008-f873-4d22-83d8-aaf31d2a9cb8" (UID: "31dbc008-f873-4d22-83d8-aaf31d2a9cb8"). InnerVolumeSpecName "kube-api-access-6ngt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:38:56.152564 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:56.152539    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ngt8\" (UniqueName: \"kubernetes.io/projected/31dbc008-f873-4d22-83d8-aaf31d2a9cb8-kube-api-access-6ngt8\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:38:56.152564 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:56.152562    4393 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/31dbc008-f873-4d22-83d8-aaf31d2a9cb8-extensions-socket-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:38:56.509529 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:38:56.509496    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:38:56.698956 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:56.698924    4393 generic.go:356] "Generic (PLEG): container finished" podID="31dbc008-f873-4d22-83d8-aaf31d2a9cb8" containerID="7f7aaf45321a74eee8d5a4f7469c4f1c815a9483f04679403d02d0938f6d8907" exitCode=0
Oct 30 00:38:56.699318 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:56.698976    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wgfbs"
Oct 30 00:38:56.699318 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:56.699005    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wgfbs" event={"ID":"31dbc008-f873-4d22-83d8-aaf31d2a9cb8","Type":"ContainerDied","Data":"7f7aaf45321a74eee8d5a4f7469c4f1c815a9483f04679403d02d0938f6d8907"}
Oct 30 00:38:56.699318 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:56.699058    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wgfbs" event={"ID":"31dbc008-f873-4d22-83d8-aaf31d2a9cb8","Type":"ContainerDied","Data":"29ddcb548262cec435bbf5ff6c93c2011e171b704c04c318109e5c59c303b2e8"}
Oct 30 00:38:56.699318 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:56.699080    4393 scope.go:122] "RemoveContainer" containerID="7f7aaf45321a74eee8d5a4f7469c4f1c815a9483f04679403d02d0938f6d8907"
Oct 30 00:38:56.707973 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:56.707959    4393 scope.go:122] "RemoveContainer" containerID="7f7aaf45321a74eee8d5a4f7469c4f1c815a9483f04679403d02d0938f6d8907"
Oct 30 00:38:56.708251 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:38:56.708233    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f7aaf45321a74eee8d5a4f7469c4f1c815a9483f04679403d02d0938f6d8907\": container with ID starting with 7f7aaf45321a74eee8d5a4f7469c4f1c815a9483f04679403d02d0938f6d8907 not found: ID does not exist" containerID="7f7aaf45321a74eee8d5a4f7469c4f1c815a9483f04679403d02d0938f6d8907"
Oct 30 00:38:56.708303 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:56.708261    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f7aaf45321a74eee8d5a4f7469c4f1c815a9483f04679403d02d0938f6d8907"} err="failed to get container status \"7f7aaf45321a74eee8d5a4f7469c4f1c815a9483f04679403d02d0938f6d8907\": rpc error: code = NotFound desc = could not find container \"7f7aaf45321a74eee8d5a4f7469c4f1c815a9483f04679403d02d0938f6d8907\": container with ID starting with 7f7aaf45321a74eee8d5a4f7469c4f1c815a9483f04679403d02d0938f6d8907 not found: ID does not exist"
Oct 30 00:38:56.753059 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:56.753031    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wgfbs"]
Oct 30 00:38:56.764513 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:56.764452    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wgfbs"]
Oct 30 00:38:57.721890 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:38:57.721857    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31dbc008-f873-4d22-83d8-aaf31d2a9cb8" path="/var/lib/kubelet/pods/31dbc008-f873-4d22-83d8-aaf31d2a9cb8/volumes"
Oct 30 00:39:12.504023 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:12.503992    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-sppzp"]
Oct 30 00:39:12.504618 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:12.504217    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-sppzp" podUID="0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43" containerName="manager" containerID="cri-o://072fe61d9937e495eec008a239aa500f1343807e6898add130d24276001988b8" gracePeriod=10
Oct 30 00:39:12.929173 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:12.929144    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"]
Oct 30 00:39:12.929681 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:12.929650    4393 cpu_manager.go:414] "containerMap: removing container" podUID="31dbc008-f873-4d22-83d8-aaf31d2a9cb8" containerName="manager"
Oct 30 00:39:12.929681 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:12.929667    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31dbc008-f873-4d22-83d8-aaf31d2a9cb8" containerName="manager"
Oct 30 00:39:12.929883 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:12.929773    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="31dbc008-f873-4d22-83d8-aaf31d2a9cb8" containerName="manager"
Oct 30 00:39:12.931959 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:12.931941    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:39:12.956749 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:12.956716    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"]
Oct 30 00:39:12.957140 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:12.957114    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gwncz\""
Oct 30 00:39:13.005397 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.005358    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3bd5875-a930-4985-b820-efe272d30763-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-cngvb\" (UID: \"f3bd5875-a930-4985-b820-efe272d30763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:39:13.005598 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.005491    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3bd5875-a930-4985-b820-efe272d30763-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-cngvb\" (UID: \"f3bd5875-a930-4985-b820-efe272d30763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:39:13.005598 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.005527    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3bd5875-a930-4985-b820-efe272d30763-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-cngvb\" (UID: \"f3bd5875-a930-4985-b820-efe272d30763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:39:13.005598 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.005566    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3bd5875-a930-4985-b820-efe272d30763-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-cngvb\" (UID: \"f3bd5875-a930-4985-b820-efe272d30763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:39:13.005853 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.005622    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3bd5875-a930-4985-b820-efe272d30763-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-cngvb\" (UID: \"f3bd5875-a930-4985-b820-efe272d30763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:39:13.005853 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.005651    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkrmk\" (UniqueName: \"kubernetes.io/projected/f3bd5875-a930-4985-b820-efe272d30763-kube-api-access-hkrmk\") pod \"maas-default-gateway-openshift-default-7d9897b54c-cngvb\" (UID: \"f3bd5875-a930-4985-b820-efe272d30763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:39:13.005853 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.005681    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3bd5875-a930-4985-b820-efe272d30763-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-cngvb\" (UID: \"f3bd5875-a930-4985-b820-efe272d30763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:39:13.005853 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.005731    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3bd5875-a930-4985-b820-efe272d30763-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-cngvb\" (UID: \"f3bd5875-a930-4985-b820-efe272d30763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:39:13.005853 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.005811    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3bd5875-a930-4985-b820-efe272d30763-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-cngvb\" (UID: \"f3bd5875-a930-4985-b820-efe272d30763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:39:13.067982 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.067955    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-sppzp"
Oct 30 00:39:13.106458 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.106426    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3bd5875-a930-4985-b820-efe272d30763-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-cngvb\" (UID: \"f3bd5875-a930-4985-b820-efe272d30763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:39:13.106627 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.106476    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3bd5875-a930-4985-b820-efe272d30763-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-cngvb\" (UID: \"f3bd5875-a930-4985-b820-efe272d30763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:39:13.106627 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.106515    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3bd5875-a930-4985-b820-efe272d30763-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-cngvb\" (UID: \"f3bd5875-a930-4985-b820-efe272d30763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:39:13.106627 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.106568    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3bd5875-a930-4985-b820-efe272d30763-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-cngvb\" (UID: \"f3bd5875-a930-4985-b820-efe272d30763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:39:13.106627 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.106590    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3bd5875-a930-4985-b820-efe272d30763-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-cngvb\" (UID: \"f3bd5875-a930-4985-b820-efe272d30763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:39:13.106627 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.106620    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3bd5875-a930-4985-b820-efe272d30763-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-cngvb\" (UID: \"f3bd5875-a930-4985-b820-efe272d30763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:39:13.106938 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.106686    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3bd5875-a930-4985-b820-efe272d30763-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-cngvb\" (UID: \"f3bd5875-a930-4985-b820-efe272d30763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:39:13.106938 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.106845    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkrmk\" (UniqueName: \"kubernetes.io/projected/f3bd5875-a930-4985-b820-efe272d30763-kube-api-access-hkrmk\") pod \"maas-default-gateway-openshift-default-7d9897b54c-cngvb\" (UID: \"f3bd5875-a930-4985-b820-efe272d30763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:39:13.106938 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.106914    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3bd5875-a930-4985-b820-efe272d30763-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-cngvb\" (UID: \"f3bd5875-a930-4985-b820-efe272d30763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:39:13.107151 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.107019    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3bd5875-a930-4985-b820-efe272d30763-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-cngvb\" (UID: \"f3bd5875-a930-4985-b820-efe272d30763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:39:13.107264 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.107242    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3bd5875-a930-4985-b820-efe272d30763-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-cngvb\" (UID: \"f3bd5875-a930-4985-b820-efe272d30763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:39:13.107404 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.107385    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3bd5875-a930-4985-b820-efe272d30763-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-cngvb\" (UID: \"f3bd5875-a930-4985-b820-efe272d30763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:39:13.107475 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.107434    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3bd5875-a930-4985-b820-efe272d30763-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-cngvb\" (UID: \"f3bd5875-a930-4985-b820-efe272d30763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:39:13.107475 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.107444    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3bd5875-a930-4985-b820-efe272d30763-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-cngvb\" (UID: \"f3bd5875-a930-4985-b820-efe272d30763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:39:13.109626 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.109606    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3bd5875-a930-4985-b820-efe272d30763-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-cngvb\" (UID: \"f3bd5875-a930-4985-b820-efe272d30763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:39:13.109948 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.109929    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3bd5875-a930-4985-b820-efe272d30763-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-cngvb\" (UID: \"f3bd5875-a930-4985-b820-efe272d30763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:39:13.138624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.138596    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3bd5875-a930-4985-b820-efe272d30763-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-cngvb\" (UID: \"f3bd5875-a930-4985-b820-efe272d30763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:39:13.138763 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.138648    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkrmk\" (UniqueName: \"kubernetes.io/projected/f3bd5875-a930-4985-b820-efe272d30763-kube-api-access-hkrmk\") pod \"maas-default-gateway-openshift-default-7d9897b54c-cngvb\" (UID: \"f3bd5875-a930-4985-b820-efe272d30763\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:39:13.208110 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.208035    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43-extensions-socket-volume\") pod \"0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43\" (UID: \"0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43\") "
Oct 30 00:39:13.208110 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.208080    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43-kube-api-access-86hlj\" (UniqueName: \"kubernetes.io/projected/0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43-kube-api-access-86hlj\") pod \"0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43\" (UID: \"0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43\") "
Oct 30 00:39:13.208491 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.208462    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43-extensions-socket-volume" pod "0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43" (UID: "0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:39:13.210526 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.210496    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43-kube-api-access-86hlj" pod "0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43" (UID: "0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43"). InnerVolumeSpecName "kube-api-access-86hlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:39:13.248746 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.248711    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:39:13.308844 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.308811    4393 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43-extensions-socket-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:39:13.308961 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.308847    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86hlj\" (UniqueName: \"kubernetes.io/projected/0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43-kube-api-access-86hlj\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:39:13.561694 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.561662    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"]
Oct 30 00:39:13.563081 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:39:13.563052    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3bd5875_a930_4985_b820_efe272d30763.slice/crio-1dfabf26ef28b8e28d8922ab5bcdc92bf61a30db1d53f8a1e6ac6de2fbe9dc25 WatchSource:0}: Error finding container 1dfabf26ef28b8e28d8922ab5bcdc92bf61a30db1d53f8a1e6ac6de2fbe9dc25: Status 404 returned error can't find the container with id 1dfabf26ef28b8e28d8922ab5bcdc92bf61a30db1d53f8a1e6ac6de2fbe9dc25
Oct 30 00:39:13.565375 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.565340    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:39:13.565501 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.565422    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:39:13.565501 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.565470    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:39:13.797874 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.797843    4393 generic.go:356] "Generic (PLEG): container finished" podID="0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43" containerID="072fe61d9937e495eec008a239aa500f1343807e6898add130d24276001988b8" exitCode=0
Oct 30 00:39:13.798038 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.797949    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-sppzp" event={"ID":"0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43","Type":"ContainerDied","Data":"072fe61d9937e495eec008a239aa500f1343807e6898add130d24276001988b8"}
Oct 30 00:39:13.798038 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.797992    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-sppzp" event={"ID":"0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43","Type":"ContainerDied","Data":"7cbfcb012e93a0a2f552fa6984e40c55dc781ca21d459f4665510d299bf6e36d"}
Oct 30 00:39:13.798038 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.798014    4393 scope.go:122] "RemoveContainer" containerID="072fe61d9937e495eec008a239aa500f1343807e6898add130d24276001988b8"
Oct 30 00:39:13.798132 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.798122    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-sppzp"
Oct 30 00:39:13.802054 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.802023    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb" event={"ID":"f3bd5875-a930-4985-b820-efe272d30763","Type":"ContainerStarted","Data":"802b9262dabe8fb2fd4dd6e0b470fb9339285ff8c4cb9ccb5ffbafdf132530ea"}
Oct 30 00:39:13.802164 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.802080    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb" event={"ID":"f3bd5875-a930-4985-b820-efe272d30763","Type":"ContainerStarted","Data":"1dfabf26ef28b8e28d8922ab5bcdc92bf61a30db1d53f8a1e6ac6de2fbe9dc25"}
Oct 30 00:39:13.808507 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.808486    4393 scope.go:122] "RemoveContainer" containerID="072fe61d9937e495eec008a239aa500f1343807e6898add130d24276001988b8"
Oct 30 00:39:13.808759 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:39:13.808739    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"072fe61d9937e495eec008a239aa500f1343807e6898add130d24276001988b8\": container with ID starting with 072fe61d9937e495eec008a239aa500f1343807e6898add130d24276001988b8 not found: ID does not exist" containerID="072fe61d9937e495eec008a239aa500f1343807e6898add130d24276001988b8"
Oct 30 00:39:13.808836 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.808770    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"072fe61d9937e495eec008a239aa500f1343807e6898add130d24276001988b8"} err="failed to get container status \"072fe61d9937e495eec008a239aa500f1343807e6898add130d24276001988b8\": rpc error: code = NotFound desc = could not find container \"072fe61d9937e495eec008a239aa500f1343807e6898add130d24276001988b8\": container with ID starting with 072fe61d9937e495eec008a239aa500f1343807e6898add130d24276001988b8 not found: ID does not exist"
Oct 30 00:39:13.853243 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.853166    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-sppzp"]
Oct 30 00:39:13.865066 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.865024    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-sppzp"]
Oct 30 00:39:13.913010 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:13.912953    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb" podStartSLOduration=1.9129367849999999 podStartE2EDuration="1.912936785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:39:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:39:13.910620289 +0000 UTC m=+444.298296680" watchObservedRunningTime="2026-10-30 00:39:13.912936785 +0000 UTC m=+444.300613220"
Oct 30 00:39:14.249941 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:14.249906    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:39:14.255366 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:14.255339    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:39:14.808042 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:14.808006    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:39:14.809099 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:14.809080    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:39:15.723149 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:15.723112    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43" path="/var/lib/kubelet/pods/0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43/volumes"
Oct 30 00:39:26.508817 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:39:26.508763    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:39:30.503071 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:30.503036    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wdr65"]
Oct 30 00:39:30.503530 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:30.503412    4393 cpu_manager.go:414] "containerMap: removing container" podUID="0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43" containerName="manager"
Oct 30 00:39:30.503530 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:30.503423    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43" containerName="manager"
Oct 30 00:39:30.503530 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:30.503511    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dd5ac6a-e3d7-41e2-aaf8-57a149ecfd43" containerName="manager"
Oct 30 00:39:30.506079 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:30.506062    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-wdr65"
Oct 30 00:39:30.539996 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:30.539968    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 00:39:30.539996 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:30.539992    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-tt5qh\""
Oct 30 00:39:30.543294 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:30.543268    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wdr65"]
Oct 30 00:39:30.628766 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:30.628723    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wdr65"]
Oct 30 00:39:30.662903 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:30.662874    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/60ad6765-add4-4ba7-91d8-af7f3dbe5f01-config-file\") pod \"limitador-limitador-74dfcfb894-wdr65\" (UID: \"60ad6765-add4-4ba7-91d8-af7f3dbe5f01\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wdr65"
Oct 30 00:39:30.663080 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:30.662914    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lthv2\" (UniqueName: \"kubernetes.io/projected/60ad6765-add4-4ba7-91d8-af7f3dbe5f01-kube-api-access-lthv2\") pod \"limitador-limitador-74dfcfb894-wdr65\" (UID: \"60ad6765-add4-4ba7-91d8-af7f3dbe5f01\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wdr65"
Oct 30 00:39:30.764235 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:30.764129    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/60ad6765-add4-4ba7-91d8-af7f3dbe5f01-config-file\") pod \"limitador-limitador-74dfcfb894-wdr65\" (UID: \"60ad6765-add4-4ba7-91d8-af7f3dbe5f01\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wdr65"
Oct 30 00:39:30.764235 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:30.764202    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lthv2\" (UniqueName: \"kubernetes.io/projected/60ad6765-add4-4ba7-91d8-af7f3dbe5f01-kube-api-access-lthv2\") pod \"limitador-limitador-74dfcfb894-wdr65\" (UID: \"60ad6765-add4-4ba7-91d8-af7f3dbe5f01\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wdr65"
Oct 30 00:39:30.764830 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:30.764805    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/60ad6765-add4-4ba7-91d8-af7f3dbe5f01-config-file\") pod \"limitador-limitador-74dfcfb894-wdr65\" (UID: \"60ad6765-add4-4ba7-91d8-af7f3dbe5f01\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wdr65"
Oct 30 00:39:30.787400 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:30.787368    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lthv2\" (UniqueName: \"kubernetes.io/projected/60ad6765-add4-4ba7-91d8-af7f3dbe5f01-kube-api-access-lthv2\") pod \"limitador-limitador-74dfcfb894-wdr65\" (UID: \"60ad6765-add4-4ba7-91d8-af7f3dbe5f01\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wdr65"
Oct 30 00:39:30.817119 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:30.817087    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-wdr65"
Oct 30 00:39:31.114019 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:31.113989    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wdr65"]
Oct 30 00:39:31.115798 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:39:31.115759    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60ad6765_add4_4ba7_91d8_af7f3dbe5f01.slice/crio-f1cde39a943fd557ec126086a2238169caa2196ab6c92a47a98749ba21394058 WatchSource:0}: Error finding container f1cde39a943fd557ec126086a2238169caa2196ab6c92a47a98749ba21394058: Status 404 returned error can't find the container with id f1cde39a943fd557ec126086a2238169caa2196ab6c92a47a98749ba21394058
Oct 30 00:39:31.893504 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:31.893465    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-wdr65" event={"ID":"60ad6765-add4-4ba7-91d8-af7f3dbe5f01","Type":"ContainerStarted","Data":"f1cde39a943fd557ec126086a2238169caa2196ab6c92a47a98749ba21394058"}
Oct 30 00:39:32.899755 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:32.899717    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-wdr65" event={"ID":"60ad6765-add4-4ba7-91d8-af7f3dbe5f01","Type":"ContainerStarted","Data":"840f078a9f439d16f654e8dda7267320d83356b4b95a981a54d27117a8f0cdc9"}
Oct 30 00:39:32.900156 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:32.899873    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-wdr65"
Oct 30 00:39:32.939919 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:32.939864    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-wdr65" podStartSLOduration=2.105424736 podStartE2EDuration="2.939851015s" totalImagesPullingTime="834.426279ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:39:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:39:32.938585543 +0000 UTC m=+463.326261936" watchObservedRunningTime="2026-10-30 00:39:32.939851015 +0000 UTC m=+463.327527404"
Oct 30 00:39:43.905492 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:43.905452    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-wdr65"
Oct 30 00:39:45.598007 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:45.597961    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wdr65"]
Oct 30 00:39:45.598436 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:45.598203    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-wdr65" podUID="60ad6765-add4-4ba7-91d8-af7f3dbe5f01" containerName="limitador" containerID="cri-o://840f078a9f439d16f654e8dda7267320d83356b4b95a981a54d27117a8f0cdc9" gracePeriod=30
Oct 30 00:39:46.143044 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:46.143019    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-wdr65"
Oct 30 00:39:46.200560 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:46.200484    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60ad6765-add4-4ba7-91d8-af7f3dbe5f01-kube-api-access-lthv2\" (UniqueName: \"kubernetes.io/projected/60ad6765-add4-4ba7-91d8-af7f3dbe5f01-kube-api-access-lthv2\") pod \"60ad6765-add4-4ba7-91d8-af7f3dbe5f01\" (UID: \"60ad6765-add4-4ba7-91d8-af7f3dbe5f01\") "
Oct 30 00:39:46.200701 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:46.200605    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60ad6765-add4-4ba7-91d8-af7f3dbe5f01-config-file\" (UniqueName: \"kubernetes.io/configmap/60ad6765-add4-4ba7-91d8-af7f3dbe5f01-config-file\") pod \"60ad6765-add4-4ba7-91d8-af7f3dbe5f01\" (UID: \"60ad6765-add4-4ba7-91d8-af7f3dbe5f01\") "
Oct 30 00:39:46.200960 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:46.200933    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60ad6765-add4-4ba7-91d8-af7f3dbe5f01-config-file" pod "60ad6765-add4-4ba7-91d8-af7f3dbe5f01" (UID: "60ad6765-add4-4ba7-91d8-af7f3dbe5f01"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:39:46.202754 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:46.202722    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60ad6765-add4-4ba7-91d8-af7f3dbe5f01-kube-api-access-lthv2" pod "60ad6765-add4-4ba7-91d8-af7f3dbe5f01" (UID: "60ad6765-add4-4ba7-91d8-af7f3dbe5f01"). InnerVolumeSpecName "kube-api-access-lthv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:39:46.301719 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:46.301675    4393 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/60ad6765-add4-4ba7-91d8-af7f3dbe5f01-config-file\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:39:46.301719 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:46.301717    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lthv2\" (UniqueName: \"kubernetes.io/projected/60ad6765-add4-4ba7-91d8-af7f3dbe5f01-kube-api-access-lthv2\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:39:46.968939 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:46.968901    4393 generic.go:356] "Generic (PLEG): container finished" podID="60ad6765-add4-4ba7-91d8-af7f3dbe5f01" containerID="840f078a9f439d16f654e8dda7267320d83356b4b95a981a54d27117a8f0cdc9" exitCode=0
Oct 30 00:39:46.969443 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:46.968946    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-wdr65" event={"ID":"60ad6765-add4-4ba7-91d8-af7f3dbe5f01","Type":"ContainerDied","Data":"840f078a9f439d16f654e8dda7267320d83356b4b95a981a54d27117a8f0cdc9"}
Oct 30 00:39:46.969443 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:46.968956    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-wdr65"
Oct 30 00:39:46.969443 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:46.968987    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-wdr65" event={"ID":"60ad6765-add4-4ba7-91d8-af7f3dbe5f01","Type":"ContainerDied","Data":"f1cde39a943fd557ec126086a2238169caa2196ab6c92a47a98749ba21394058"}
Oct 30 00:39:46.969443 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:46.969008    4393 scope.go:122] "RemoveContainer" containerID="840f078a9f439d16f654e8dda7267320d83356b4b95a981a54d27117a8f0cdc9"
Oct 30 00:39:46.978561 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:46.978545    4393 scope.go:122] "RemoveContainer" containerID="840f078a9f439d16f654e8dda7267320d83356b4b95a981a54d27117a8f0cdc9"
Oct 30 00:39:46.978861 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:39:46.978844    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"840f078a9f439d16f654e8dda7267320d83356b4b95a981a54d27117a8f0cdc9\": container with ID starting with 840f078a9f439d16f654e8dda7267320d83356b4b95a981a54d27117a8f0cdc9 not found: ID does not exist" containerID="840f078a9f439d16f654e8dda7267320d83356b4b95a981a54d27117a8f0cdc9"
Oct 30 00:39:46.978902 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:46.978871    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"840f078a9f439d16f654e8dda7267320d83356b4b95a981a54d27117a8f0cdc9"} err="failed to get container status \"840f078a9f439d16f654e8dda7267320d83356b4b95a981a54d27117a8f0cdc9\": rpc error: code = NotFound desc = could not find container \"840f078a9f439d16f654e8dda7267320d83356b4b95a981a54d27117a8f0cdc9\": container with ID starting with 840f078a9f439d16f654e8dda7267320d83356b4b95a981a54d27117a8f0cdc9 not found: ID does not exist"
Oct 30 00:39:47.018893 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:47.018860    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wdr65"]
Oct 30 00:39:47.030475 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:47.030447    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wdr65"]
Oct 30 00:39:47.723000 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:47.722965    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60ad6765-add4-4ba7-91d8-af7f3dbe5f01" path="/var/lib/kubelet/pods/60ad6765-add4-4ba7-91d8-af7f3dbe5f01/volumes"
Oct 30 00:39:47.969503 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:47.969461    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-qzr7h"]
Oct 30 00:39:47.969933 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:47.969886    4393 cpu_manager.go:414] "containerMap: removing container" podUID="60ad6765-add4-4ba7-91d8-af7f3dbe5f01" containerName="limitador"
Oct 30 00:39:47.969933 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:47.969898    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60ad6765-add4-4ba7-91d8-af7f3dbe5f01" containerName="limitador"
Oct 30 00:39:47.970006 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:47.969967    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="60ad6765-add4-4ba7-91d8-af7f3dbe5f01" containerName="limitador"
Oct 30 00:39:47.972498 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:47.972478    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qzr7h"
Oct 30 00:39:47.987063 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:47.987044    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Oct 30 00:39:47.990146 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:47.990125    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-qlz4p\""
Oct 30 00:39:48.001251 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:48.001230    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-qzr7h"]
Oct 30 00:39:48.001496 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:48.001475    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 00:39:48.001600 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:48.001486    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 00:39:48.017343 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:48.017317    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/42437b4a-aa45-4add-b6ad-e774be593b64-data\") pod \"postgres-7cdbf6dc44-qzr7h\" (UID: \"42437b4a-aa45-4add-b6ad-e774be593b64\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qzr7h"
Oct 30 00:39:48.017449 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:48.017356    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6jqp\" (UniqueName: \"kubernetes.io/projected/42437b4a-aa45-4add-b6ad-e774be593b64-kube-api-access-q6jqp\") pod \"postgres-7cdbf6dc44-qzr7h\" (UID: \"42437b4a-aa45-4add-b6ad-e774be593b64\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qzr7h"
Oct 30 00:39:48.118856 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:48.118823    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6jqp\" (UniqueName: \"kubernetes.io/projected/42437b4a-aa45-4add-b6ad-e774be593b64-kube-api-access-q6jqp\") pod \"postgres-7cdbf6dc44-qzr7h\" (UID: \"42437b4a-aa45-4add-b6ad-e774be593b64\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qzr7h"
Oct 30 00:39:48.119015 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:48.118930    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/42437b4a-aa45-4add-b6ad-e774be593b64-data\") pod \"postgres-7cdbf6dc44-qzr7h\" (UID: \"42437b4a-aa45-4add-b6ad-e774be593b64\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qzr7h"
Oct 30 00:39:48.119265 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:48.119250    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/42437b4a-aa45-4add-b6ad-e774be593b64-data\") pod \"postgres-7cdbf6dc44-qzr7h\" (UID: \"42437b4a-aa45-4add-b6ad-e774be593b64\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qzr7h"
Oct 30 00:39:48.141815 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:48.141787    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6jqp\" (UniqueName: \"kubernetes.io/projected/42437b4a-aa45-4add-b6ad-e774be593b64-kube-api-access-q6jqp\") pod \"postgres-7cdbf6dc44-qzr7h\" (UID: \"42437b4a-aa45-4add-b6ad-e774be593b64\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qzr7h"
Oct 30 00:39:48.284041 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:48.283966    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qzr7h"
Oct 30 00:39:48.579060 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:48.579031    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-qzr7h"]
Oct 30 00:39:48.580558 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:39:48.580527    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42437b4a_aa45_4add_b6ad_e774be593b64.slice/crio-62b67dc5a049ced172064f222439160efd4969ee226c30ab621933ab76bf06f1 WatchSource:0}: Error finding container 62b67dc5a049ced172064f222439160efd4969ee226c30ab621933ab76bf06f1: Status 404 returned error can't find the container with id 62b67dc5a049ced172064f222439160efd4969ee226c30ab621933ab76bf06f1
Oct 30 00:39:48.982062 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:48.982026    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qzr7h" event={"ID":"42437b4a-aa45-4add-b6ad-e774be593b64","Type":"ContainerStarted","Data":"62b67dc5a049ced172064f222439160efd4969ee226c30ab621933ab76bf06f1"}
Oct 30 00:39:53.199685 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:53.199655    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Oct 30 00:39:54.013522 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:54.013485    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qzr7h" event={"ID":"42437b4a-aa45-4add-b6ad-e774be593b64","Type":"ContainerStarted","Data":"23833a2d067c2c9ad23bfebc4294e90df74723f6860ab338658a8a0a6fb2713b"}
Oct 30 00:39:54.013712 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:54.013544    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qzr7h"
Oct 30 00:39:54.054052 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:39:54.054002    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qzr7h" podStartSLOduration=2.449792136 podStartE2EDuration="7.053988718s" totalImagesPullingTime="4.604196582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:39:54.052245303 +0000 UTC m=+484.439921694" watchObservedRunningTime="2026-10-30 00:39:54.053988718 +0000 UTC m=+484.441665107"
Oct 30 00:39:56.509234 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:39:56.509188    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:40:00.048101 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:00.048072    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-qzr7h"
Oct 30 00:40:09.881935 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:09.881859    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-577f596567-2wld6"]
Oct 30 00:40:09.886767 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:09.886736    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-2wld6"
Oct 30 00:40:09.908530 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:09.908507    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 00:40:09.914729 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:09.914704    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-2wld6"]
Oct 30 00:40:09.915766 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:09.915746    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vzmlw\""
Oct 30 00:40:09.915900 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:09.915833    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 00:40:09.920206 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:09.920188    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 00:40:10.011973 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:10.011943    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a316192e-fcc6-483f-8fb7-4eb53b20d505-webhook-cert\") pod \"maas-controller-577f596567-2wld6\" (UID: \"a316192e-fcc6-483f-8fb7-4eb53b20d505\") " pod="opendatahub/maas-controller-577f596567-2wld6"
Oct 30 00:40:10.012146 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:10.011981    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a316192e-fcc6-483f-8fb7-4eb53b20d505-metrics-certs\") pod \"maas-controller-577f596567-2wld6\" (UID: \"a316192e-fcc6-483f-8fb7-4eb53b20d505\") " pod="opendatahub/maas-controller-577f596567-2wld6"
Oct 30 00:40:10.012146 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:10.012066    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf5s5\" (UniqueName: \"kubernetes.io/projected/a316192e-fcc6-483f-8fb7-4eb53b20d505-kube-api-access-zf5s5\") pod \"maas-controller-577f596567-2wld6\" (UID: \"a316192e-fcc6-483f-8fb7-4eb53b20d505\") " pod="opendatahub/maas-controller-577f596567-2wld6"
Oct 30 00:40:10.112524 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:10.112496    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf5s5\" (UniqueName: \"kubernetes.io/projected/a316192e-fcc6-483f-8fb7-4eb53b20d505-kube-api-access-zf5s5\") pod \"maas-controller-577f596567-2wld6\" (UID: \"a316192e-fcc6-483f-8fb7-4eb53b20d505\") " pod="opendatahub/maas-controller-577f596567-2wld6"
Oct 30 00:40:10.112708 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:10.112578    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a316192e-fcc6-483f-8fb7-4eb53b20d505-webhook-cert\") pod \"maas-controller-577f596567-2wld6\" (UID: \"a316192e-fcc6-483f-8fb7-4eb53b20d505\") " pod="opendatahub/maas-controller-577f596567-2wld6"
Oct 30 00:40:10.112708 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:10.112603    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a316192e-fcc6-483f-8fb7-4eb53b20d505-metrics-certs\") pod \"maas-controller-577f596567-2wld6\" (UID: \"a316192e-fcc6-483f-8fb7-4eb53b20d505\") " pod="opendatahub/maas-controller-577f596567-2wld6"
Oct 30 00:40:10.115165 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:10.115135    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a316192e-fcc6-483f-8fb7-4eb53b20d505-metrics-certs\") pod \"maas-controller-577f596567-2wld6\" (UID: \"a316192e-fcc6-483f-8fb7-4eb53b20d505\") " pod="opendatahub/maas-controller-577f596567-2wld6"
Oct 30 00:40:10.115165 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:10.115160    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a316192e-fcc6-483f-8fb7-4eb53b20d505-webhook-cert\") pod \"maas-controller-577f596567-2wld6\" (UID: \"a316192e-fcc6-483f-8fb7-4eb53b20d505\") " pod="opendatahub/maas-controller-577f596567-2wld6"
Oct 30 00:40:10.135400 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:10.135342    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf5s5\" (UniqueName: \"kubernetes.io/projected/a316192e-fcc6-483f-8fb7-4eb53b20d505-kube-api-access-zf5s5\") pod \"maas-controller-577f596567-2wld6\" (UID: \"a316192e-fcc6-483f-8fb7-4eb53b20d505\") " pod="opendatahub/maas-controller-577f596567-2wld6"
Oct 30 00:40:10.198507 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:10.198474    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-2wld6"
Oct 30 00:40:10.218708 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:10.218678    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6b5b68ff8f-v68v9"]
Oct 30 00:40:10.221249 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:10.221228    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b5b68ff8f-v68v9"
Oct 30 00:40:10.242830 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:10.242803    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b5b68ff8f-v68v9"]
Oct 30 00:40:10.315220 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:10.315180    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7df9692e-c210-4300-8bcd-6fc93f7dd2fe-metrics-certs\") pod \"maas-controller-6b5b68ff8f-v68v9\" (UID: \"7df9692e-c210-4300-8bcd-6fc93f7dd2fe\") " pod="opendatahub/maas-controller-6b5b68ff8f-v68v9"
Oct 30 00:40:10.315389 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:10.315304    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwcqc\" (UniqueName: \"kubernetes.io/projected/7df9692e-c210-4300-8bcd-6fc93f7dd2fe-kube-api-access-mwcqc\") pod \"maas-controller-6b5b68ff8f-v68v9\" (UID: \"7df9692e-c210-4300-8bcd-6fc93f7dd2fe\") " pod="opendatahub/maas-controller-6b5b68ff8f-v68v9"
Oct 30 00:40:10.315459 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:10.315399    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7df9692e-c210-4300-8bcd-6fc93f7dd2fe-webhook-cert\") pod \"maas-controller-6b5b68ff8f-v68v9\" (UID: \"7df9692e-c210-4300-8bcd-6fc93f7dd2fe\") " pod="opendatahub/maas-controller-6b5b68ff8f-v68v9"
Oct 30 00:40:10.416403 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:10.416320    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7df9692e-c210-4300-8bcd-6fc93f7dd2fe-webhook-cert\") pod \"maas-controller-6b5b68ff8f-v68v9\" (UID: \"7df9692e-c210-4300-8bcd-6fc93f7dd2fe\") " pod="opendatahub/maas-controller-6b5b68ff8f-v68v9"
Oct 30 00:40:10.416403 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:10.416365    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7df9692e-c210-4300-8bcd-6fc93f7dd2fe-metrics-certs\") pod \"maas-controller-6b5b68ff8f-v68v9\" (UID: \"7df9692e-c210-4300-8bcd-6fc93f7dd2fe\") " pod="opendatahub/maas-controller-6b5b68ff8f-v68v9"
Oct 30 00:40:10.416403 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:10.416402    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwcqc\" (UniqueName: \"kubernetes.io/projected/7df9692e-c210-4300-8bcd-6fc93f7dd2fe-kube-api-access-mwcqc\") pod \"maas-controller-6b5b68ff8f-v68v9\" (UID: \"7df9692e-c210-4300-8bcd-6fc93f7dd2fe\") " pod="opendatahub/maas-controller-6b5b68ff8f-v68v9"
Oct 30 00:40:10.419049 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:10.419025    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7df9692e-c210-4300-8bcd-6fc93f7dd2fe-webhook-cert\") pod \"maas-controller-6b5b68ff8f-v68v9\" (UID: \"7df9692e-c210-4300-8bcd-6fc93f7dd2fe\") " pod="opendatahub/maas-controller-6b5b68ff8f-v68v9"
Oct 30 00:40:10.419174 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:10.419059    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7df9692e-c210-4300-8bcd-6fc93f7dd2fe-metrics-certs\") pod \"maas-controller-6b5b68ff8f-v68v9\" (UID: \"7df9692e-c210-4300-8bcd-6fc93f7dd2fe\") " pod="opendatahub/maas-controller-6b5b68ff8f-v68v9"
Oct 30 00:40:10.439544 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:10.439510    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwcqc\" (UniqueName: \"kubernetes.io/projected/7df9692e-c210-4300-8bcd-6fc93f7dd2fe-kube-api-access-mwcqc\") pod \"maas-controller-6b5b68ff8f-v68v9\" (UID: \"7df9692e-c210-4300-8bcd-6fc93f7dd2fe\") " pod="opendatahub/maas-controller-6b5b68ff8f-v68v9"
Oct 30 00:40:10.501864 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:10.501826    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-2wld6"]
Oct 30 00:40:10.508207 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:40:10.508174    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda316192e_fcc6_483f_8fb7_4eb53b20d505.slice/crio-161ffa544258e07603a8f3a888a9c2ca15637b027652b5a4dbfd00d9e8d4430f WatchSource:0}: Error finding container 161ffa544258e07603a8f3a888a9c2ca15637b027652b5a4dbfd00d9e8d4430f: Status 404 returned error can't find the container with id 161ffa544258e07603a8f3a888a9c2ca15637b027652b5a4dbfd00d9e8d4430f
Oct 30 00:40:10.533150 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:10.533117    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b5b68ff8f-v68v9"
Oct 30 00:40:10.829383 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:10.829361    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b5b68ff8f-v68v9"]
Oct 30 00:40:10.831089 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:40:10.831062    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df9692e_c210_4300_8bcd_6fc93f7dd2fe.slice/crio-0bfab7b2c6283ae2e54875f06e764b53f117c58605d3e0e62f82e5432b7129d9 WatchSource:0}: Error finding container 0bfab7b2c6283ae2e54875f06e764b53f117c58605d3e0e62f82e5432b7129d9: Status 404 returned error can't find the container with id 0bfab7b2c6283ae2e54875f06e764b53f117c58605d3e0e62f82e5432b7129d9
Oct 30 00:40:11.107648 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:11.107567    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b5b68ff8f-v68v9" event={"ID":"7df9692e-c210-4300-8bcd-6fc93f7dd2fe","Type":"ContainerStarted","Data":"0bfab7b2c6283ae2e54875f06e764b53f117c58605d3e0e62f82e5432b7129d9"}
Oct 30 00:40:11.108917 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:11.108889    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-2wld6" event={"ID":"a316192e-fcc6-483f-8fb7-4eb53b20d505","Type":"ContainerStarted","Data":"161ffa544258e07603a8f3a888a9c2ca15637b027652b5a4dbfd00d9e8d4430f"}
Oct 30 00:40:14.125632 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:14.125574    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b5b68ff8f-v68v9" event={"ID":"7df9692e-c210-4300-8bcd-6fc93f7dd2fe","Type":"ContainerStarted","Data":"7085b5c7f1aae1d2d9b384c83716f964ad3edad3555862727f11807cdf66b24a"}
Oct 30 00:40:14.126100 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:14.125686    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6b5b68ff8f-v68v9"
Oct 30 00:40:14.127334 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:14.127309    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-2wld6" event={"ID":"a316192e-fcc6-483f-8fb7-4eb53b20d505","Type":"ContainerStarted","Data":"d715239941a64f501e0270d644c702cbf2a7f2ad29cf9b80c084d256228b3c8d"}
Oct 30 00:40:14.127453 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:14.127409    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-577f596567-2wld6"
Oct 30 00:40:14.167191 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:14.167144    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6b5b68ff8f-v68v9" podStartSLOduration=1.569278111 podStartE2EDuration="4.167132548s" totalImagesPullingTime="2.597854437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:40:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:40:14.166033942 +0000 UTC m=+504.553710331" watchObservedRunningTime="2026-10-30 00:40:14.167132548 +0000 UTC m=+504.554809002"
Oct 30 00:40:14.209191 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:14.209149    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-577f596567-2wld6" podStartSLOduration=2.533775243 podStartE2EDuration="5.209135722s" totalImagesPullingTime="2.675360479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:40:14.206869205 +0000 UTC m=+504.594545595" watchObservedRunningTime="2026-10-30 00:40:14.209135722 +0000 UTC m=+504.596812141"
Oct 30 00:40:16.948303 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:16.948273    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-fn7wx"]
Oct 30 00:40:16.950853 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:16.950832    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-fn7wx"
Oct 30 00:40:16.970849 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:16.970831    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 30 00:40:16.982440 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:16.982421    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-5554v\""
Oct 30 00:40:16.982570 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:16.982554    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 30 00:40:16.982641 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:16.982587    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 30 00:40:16.986101 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:16.986077    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-fn7wx"]
Oct 30 00:40:17.077110 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:17.077077    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r24mj\" (UniqueName: \"kubernetes.io/projected/55f3efa3-7ddf-47e5-be8c-1acb8f1b6d3f-kube-api-access-r24mj\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-fn7wx\" (UID: \"55f3efa3-7ddf-47e5-be8c-1acb8f1b6d3f\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-fn7wx"
Oct 30 00:40:17.077286 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:17.077117    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55f3efa3-7ddf-47e5-be8c-1acb8f1b6d3f-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-fn7wx\" (UID: \"55f3efa3-7ddf-47e5-be8c-1acb8f1b6d3f\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-fn7wx"
Oct 30 00:40:17.077286 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:17.077222    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/55f3efa3-7ddf-47e5-be8c-1acb8f1b6d3f-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-fn7wx\" (UID: \"55f3efa3-7ddf-47e5-be8c-1acb8f1b6d3f\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-fn7wx"
Oct 30 00:40:17.077406 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:17.077378    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/55f3efa3-7ddf-47e5-be8c-1acb8f1b6d3f-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-fn7wx\" (UID: \"55f3efa3-7ddf-47e5-be8c-1acb8f1b6d3f\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-fn7wx"
Oct 30 00:40:17.178769 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:17.178736    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r24mj\" (UniqueName: \"kubernetes.io/projected/55f3efa3-7ddf-47e5-be8c-1acb8f1b6d3f-kube-api-access-r24mj\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-fn7wx\" (UID: \"55f3efa3-7ddf-47e5-be8c-1acb8f1b6d3f\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-fn7wx"
Oct 30 00:40:17.178982 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:17.178804    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55f3efa3-7ddf-47e5-be8c-1acb8f1b6d3f-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-fn7wx\" (UID: \"55f3efa3-7ddf-47e5-be8c-1acb8f1b6d3f\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-fn7wx"
Oct 30 00:40:17.178982 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:17.178848    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/55f3efa3-7ddf-47e5-be8c-1acb8f1b6d3f-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-fn7wx\" (UID: \"55f3efa3-7ddf-47e5-be8c-1acb8f1b6d3f\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-fn7wx"
Oct 30 00:40:17.178982 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:17.178902    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/55f3efa3-7ddf-47e5-be8c-1acb8f1b6d3f-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-fn7wx\" (UID: \"55f3efa3-7ddf-47e5-be8c-1acb8f1b6d3f\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-fn7wx"
Oct 30 00:40:17.179545 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:17.179521    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55f3efa3-7ddf-47e5-be8c-1acb8f1b6d3f-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-fn7wx\" (UID: \"55f3efa3-7ddf-47e5-be8c-1acb8f1b6d3f\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-fn7wx"
Oct 30 00:40:17.179672 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:17.179654    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/55f3efa3-7ddf-47e5-be8c-1acb8f1b6d3f-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-fn7wx\" (UID: \"55f3efa3-7ddf-47e5-be8c-1acb8f1b6d3f\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-fn7wx"
Oct 30 00:40:17.181380 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:17.181363    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/55f3efa3-7ddf-47e5-be8c-1acb8f1b6d3f-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-fn7wx\" (UID: \"55f3efa3-7ddf-47e5-be8c-1acb8f1b6d3f\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-fn7wx"
Oct 30 00:40:17.201635 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:17.201580    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r24mj\" (UniqueName: \"kubernetes.io/projected/55f3efa3-7ddf-47e5-be8c-1acb8f1b6d3f-kube-api-access-r24mj\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-fn7wx\" (UID: \"55f3efa3-7ddf-47e5-be8c-1acb8f1b6d3f\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-fn7wx"
Oct 30 00:40:17.262516 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:17.262484    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-fn7wx"
Oct 30 00:40:17.558647 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:17.558621    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-fn7wx"]
Oct 30 00:40:17.559733 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:40:17.559704    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f3efa3_7ddf_47e5_be8c_1acb8f1b6d3f.slice/crio-61844ec9750b68f462215604bfee005074b8c0f28692279676135760dff1960d WatchSource:0}: Error finding container 61844ec9750b68f462215604bfee005074b8c0f28692279676135760dff1960d: Status 404 returned error can't find the container with id 61844ec9750b68f462215604bfee005074b8c0f28692279676135760dff1960d
Oct 30 00:40:18.150428 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:18.150396    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-fn7wx" event={"ID":"55f3efa3-7ddf-47e5-be8c-1acb8f1b6d3f","Type":"ContainerStarted","Data":"61844ec9750b68f462215604bfee005074b8c0f28692279676135760dff1960d"}
Oct 30 00:40:25.135651 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:25.135616    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-577f596567-2wld6"
Oct 30 00:40:25.136147 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:25.135706    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6b5b68ff8f-v68v9"
Oct 30 00:40:25.250336 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:25.249545    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-2wld6"]
Oct 30 00:40:25.250336 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:25.249826    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-577f596567-2wld6" podUID="a316192e-fcc6-483f-8fb7-4eb53b20d505" containerName="manager" containerID="cri-o://d715239941a64f501e0270d644c702cbf2a7f2ad29cf9b80c084d256228b3c8d" gracePeriod=30
Oct 30 00:40:25.511880 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:25.511854    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-2wld6"
Oct 30 00:40:25.662575 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:25.662373    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a316192e-fcc6-483f-8fb7-4eb53b20d505-webhook-cert\" (UniqueName: \"kubernetes.io/secret/a316192e-fcc6-483f-8fb7-4eb53b20d505-webhook-cert\") pod \"a316192e-fcc6-483f-8fb7-4eb53b20d505\" (UID: \"a316192e-fcc6-483f-8fb7-4eb53b20d505\") "
Oct 30 00:40:25.662575 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:25.662458    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a316192e-fcc6-483f-8fb7-4eb53b20d505-kube-api-access-zf5s5\" (UniqueName: \"kubernetes.io/projected/a316192e-fcc6-483f-8fb7-4eb53b20d505-kube-api-access-zf5s5\") pod \"a316192e-fcc6-483f-8fb7-4eb53b20d505\" (UID: \"a316192e-fcc6-483f-8fb7-4eb53b20d505\") "
Oct 30 00:40:25.662575 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:25.662487    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a316192e-fcc6-483f-8fb7-4eb53b20d505-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a316192e-fcc6-483f-8fb7-4eb53b20d505-metrics-certs\") pod \"a316192e-fcc6-483f-8fb7-4eb53b20d505\" (UID: \"a316192e-fcc6-483f-8fb7-4eb53b20d505\") "
Oct 30 00:40:25.665657 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:25.665622    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a316192e-fcc6-483f-8fb7-4eb53b20d505-metrics-certs" pod "a316192e-fcc6-483f-8fb7-4eb53b20d505" (UID: "a316192e-fcc6-483f-8fb7-4eb53b20d505"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:40:25.665835 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:25.665683    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a316192e-fcc6-483f-8fb7-4eb53b20d505-kube-api-access-zf5s5" pod "a316192e-fcc6-483f-8fb7-4eb53b20d505" (UID: "a316192e-fcc6-483f-8fb7-4eb53b20d505"). InnerVolumeSpecName "kube-api-access-zf5s5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:40:25.665984 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:25.665957    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a316192e-fcc6-483f-8fb7-4eb53b20d505-webhook-cert" pod "a316192e-fcc6-483f-8fb7-4eb53b20d505" (UID: "a316192e-fcc6-483f-8fb7-4eb53b20d505"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:40:25.764208 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:25.764177    4393 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a316192e-fcc6-483f-8fb7-4eb53b20d505-webhook-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:40:25.764208 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:25.764210    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf5s5\" (UniqueName: \"kubernetes.io/projected/a316192e-fcc6-483f-8fb7-4eb53b20d505-kube-api-access-zf5s5\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:40:25.764386 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:25.764225    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a316192e-fcc6-483f-8fb7-4eb53b20d505-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:40:26.202568 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:26.202534    4393 generic.go:356] "Generic (PLEG): container finished" podID="a316192e-fcc6-483f-8fb7-4eb53b20d505" containerID="d715239941a64f501e0270d644c702cbf2a7f2ad29cf9b80c084d256228b3c8d" exitCode=0
Oct 30 00:40:26.203104 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:26.202581    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-2wld6" event={"ID":"a316192e-fcc6-483f-8fb7-4eb53b20d505","Type":"ContainerDied","Data":"d715239941a64f501e0270d644c702cbf2a7f2ad29cf9b80c084d256228b3c8d"}
Oct 30 00:40:26.203104 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:26.202596    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-2wld6"
Oct 30 00:40:26.203104 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:26.202634    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-2wld6" event={"ID":"a316192e-fcc6-483f-8fb7-4eb53b20d505","Type":"ContainerDied","Data":"161ffa544258e07603a8f3a888a9c2ca15637b027652b5a4dbfd00d9e8d4430f"}
Oct 30 00:40:26.203104 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:26.202669    4393 scope.go:122] "RemoveContainer" containerID="d715239941a64f501e0270d644c702cbf2a7f2ad29cf9b80c084d256228b3c8d"
Oct 30 00:40:26.212303 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:26.212282    4393 scope.go:122] "RemoveContainer" containerID="d715239941a64f501e0270d644c702cbf2a7f2ad29cf9b80c084d256228b3c8d"
Oct 30 00:40:26.212613 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:40:26.212581    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d715239941a64f501e0270d644c702cbf2a7f2ad29cf9b80c084d256228b3c8d\": container with ID starting with d715239941a64f501e0270d644c702cbf2a7f2ad29cf9b80c084d256228b3c8d not found: ID does not exist" containerID="d715239941a64f501e0270d644c702cbf2a7f2ad29cf9b80c084d256228b3c8d"
Oct 30 00:40:26.212712 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:26.212632    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d715239941a64f501e0270d644c702cbf2a7f2ad29cf9b80c084d256228b3c8d"} err="failed to get container status \"d715239941a64f501e0270d644c702cbf2a7f2ad29cf9b80c084d256228b3c8d\": rpc error: code = NotFound desc = could not find container \"d715239941a64f501e0270d644c702cbf2a7f2ad29cf9b80c084d256228b3c8d\": container with ID starting with d715239941a64f501e0270d644c702cbf2a7f2ad29cf9b80c084d256228b3c8d not found: ID does not exist"
Oct 30 00:40:26.254266 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:26.254230    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-2wld6"]
Oct 30 00:40:26.266137 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:26.266107    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-577f596567-2wld6"]
Oct 30 00:40:26.514257 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:40:26.514166    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:40:27.724406 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:27.724375    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a316192e-fcc6-483f-8fb7-4eb53b20d505" path="/var/lib/kubelet/pods/a316192e-fcc6-483f-8fb7-4eb53b20d505/volumes"
Oct 30 00:40:30.227964 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:30.227925    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-fn7wx" event={"ID":"55f3efa3-7ddf-47e5-be8c-1acb8f1b6d3f","Type":"ContainerStarted","Data":"7b405ca765a84297f04d09f065c3d2e381bf610ce9301885b9da7efcfcc157e9"}
Oct 30 00:40:30.228347 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:30.228007    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-fn7wx"
Oct 30 00:40:41.238643 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:41.238610    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-fn7wx"
Oct 30 00:40:41.280330 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:41.280278    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-fn7wx" podStartSLOduration=13.661855122 podStartE2EDuration="25.280264445s" totalImagesPullingTime="11.618409323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:40:30.273702755 +0000 UTC m=+520.661379144" watchObservedRunningTime="2026-10-30 00:40:41.280264445 +0000 UTC m=+531.667940833"
Oct 30 00:40:47.679073 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:47.679040    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-27r2g"]
Oct 30 00:40:47.679547 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:47.679526    4393 cpu_manager.go:414] "containerMap: removing container" podUID="a316192e-fcc6-483f-8fb7-4eb53b20d505" containerName="manager"
Oct 30 00:40:47.679547 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:47.679545    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a316192e-fcc6-483f-8fb7-4eb53b20d505" containerName="manager"
Oct 30 00:40:47.679639 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:47.679609    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="a316192e-fcc6-483f-8fb7-4eb53b20d505" containerName="manager"
Oct 30 00:40:47.686891 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:47.686869    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-27r2g"
Oct 30 00:40:47.714525 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:47.714494    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 00:40:47.714525 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:47.714524    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 00:40:47.714831 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:47.714539    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-x95xl\""
Oct 30 00:40:47.723144 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:47.723123    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-27r2g"]
Oct 30 00:40:47.757908 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:47.757879    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mkdt\" (UniqueName: \"kubernetes.io/projected/29c80ca8-5215-41d3-9c4c-a4744be8b019-kube-api-access-6mkdt\") pod \"maas-api-5c48877c75-27r2g\" (UID: \"29c80ca8-5215-41d3-9c4c-a4744be8b019\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-27r2g"
Oct 30 00:40:47.758128 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:47.757932    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29c80ca8-5215-41d3-9c4c-a4744be8b019-maas-api-tls\") pod \"maas-api-5c48877c75-27r2g\" (UID: \"29c80ca8-5215-41d3-9c4c-a4744be8b019\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-27r2g"
Oct 30 00:40:47.758128 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:47.758018    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29c80ca8-5215-41d3-9c4c-a4744be8b019-metrics-certs\") pod \"maas-api-5c48877c75-27r2g\" (UID: \"29c80ca8-5215-41d3-9c4c-a4744be8b019\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-27r2g"
Oct 30 00:40:47.758128 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:47.758060    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c80ca8-5215-41d3-9c4c-a4744be8b019-openshift-service-ca\") pod \"maas-api-5c48877c75-27r2g\" (UID: \"29c80ca8-5215-41d3-9c4c-a4744be8b019\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-27r2g"
Oct 30 00:40:47.858604 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:47.858570    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29c80ca8-5215-41d3-9c4c-a4744be8b019-metrics-certs\") pod \"maas-api-5c48877c75-27r2g\" (UID: \"29c80ca8-5215-41d3-9c4c-a4744be8b019\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-27r2g"
Oct 30 00:40:47.858801 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:47.858618    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c80ca8-5215-41d3-9c4c-a4744be8b019-openshift-service-ca\") pod \"maas-api-5c48877c75-27r2g\" (UID: \"29c80ca8-5215-41d3-9c4c-a4744be8b019\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-27r2g"
Oct 30 00:40:47.858801 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:47.858741    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mkdt\" (UniqueName: \"kubernetes.io/projected/29c80ca8-5215-41d3-9c4c-a4744be8b019-kube-api-access-6mkdt\") pod \"maas-api-5c48877c75-27r2g\" (UID: \"29c80ca8-5215-41d3-9c4c-a4744be8b019\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-27r2g"
Oct 30 00:40:47.858905 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:47.858883    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29c80ca8-5215-41d3-9c4c-a4744be8b019-maas-api-tls\") pod \"maas-api-5c48877c75-27r2g\" (UID: \"29c80ca8-5215-41d3-9c4c-a4744be8b019\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-27r2g"
Oct 30 00:40:47.859307 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:47.859284    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c80ca8-5215-41d3-9c4c-a4744be8b019-openshift-service-ca\") pod \"maas-api-5c48877c75-27r2g\" (UID: \"29c80ca8-5215-41d3-9c4c-a4744be8b019\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-27r2g"
Oct 30 00:40:47.861337 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:47.861312    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29c80ca8-5215-41d3-9c4c-a4744be8b019-maas-api-tls\") pod \"maas-api-5c48877c75-27r2g\" (UID: \"29c80ca8-5215-41d3-9c4c-a4744be8b019\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-27r2g"
Oct 30 00:40:47.861435 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:47.861353    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29c80ca8-5215-41d3-9c4c-a4744be8b019-metrics-certs\") pod \"maas-api-5c48877c75-27r2g\" (UID: \"29c80ca8-5215-41d3-9c4c-a4744be8b019\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-27r2g"
Oct 30 00:40:47.882600 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:47.882575    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mkdt\" (UniqueName: \"kubernetes.io/projected/29c80ca8-5215-41d3-9c4c-a4744be8b019-kube-api-access-6mkdt\") pod \"maas-api-5c48877c75-27r2g\" (UID: \"29c80ca8-5215-41d3-9c4c-a4744be8b019\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-27r2g"
Oct 30 00:40:48.000470 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:48.000381    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-27r2g"
Oct 30 00:40:48.298593 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:48.298571    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-27r2g"]
Oct 30 00:40:48.300471 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:40:48.300445    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c80ca8_5215_41d3_9c4c_a4744be8b019.slice/crio-2467ffdc17caa4db687d2f210b6f8adc36161beef75a42c2292042f06f034365 WatchSource:0}: Error finding container 2467ffdc17caa4db687d2f210b6f8adc36161beef75a42c2292042f06f034365: Status 404 returned error can't find the container with id 2467ffdc17caa4db687d2f210b6f8adc36161beef75a42c2292042f06f034365
Oct 30 00:40:48.317182 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:48.317153    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-27r2g" event={"ID":"29c80ca8-5215-41d3-9c4c-a4744be8b019","Type":"ContainerStarted","Data":"2467ffdc17caa4db687d2f210b6f8adc36161beef75a42c2292042f06f034365"}
Oct 30 00:40:50.337176 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:50.337141    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-27r2g" event={"ID":"29c80ca8-5215-41d3-9c4c-a4744be8b019","Type":"ContainerStarted","Data":"bab2060691968659b00faac6a74e51499b7b94d65c56523d9619dd4f02273f4b"}
Oct 30 00:40:50.337602 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:50.337252    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-27r2g"
Oct 30 00:40:50.377924 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:50.377873    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-27r2g" podStartSLOduration=1.82929995 podStartE2EDuration="3.377861684s" totalImagesPullingTime="1.548561734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:40:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:40:50.377547101 +0000 UTC m=+540.765223489" watchObservedRunningTime="2026-10-30 00:40:50.377861684 +0000 UTC m=+540.765538072"
Oct 30 00:40:56.348726 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:56.348695    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-27r2g"
Oct 30 00:40:56.512224 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:40:56.512185    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:40:59.383102 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:59.383067    4393 generic.go:356] "Generic (PLEG): container finished" podID="f3bd5875-a930-4985-b820-efe272d30763" containerID="802b9262dabe8fb2fd4dd6e0b470fb9339285ff8c4cb9ccb5ffbafdf132530ea" exitCode=0
Oct 30 00:40:59.383563 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:59.383140    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb" event={"ID":"f3bd5875-a930-4985-b820-efe272d30763","Type":"ContainerDied","Data":"802b9262dabe8fb2fd4dd6e0b470fb9339285ff8c4cb9ccb5ffbafdf132530ea"}
Oct 30 00:40:59.383563 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:59.383505    4393 scope.go:122] "RemoveContainer" containerID="802b9262dabe8fb2fd4dd6e0b470fb9339285ff8c4cb9ccb5ffbafdf132530ea"
Oct 30 00:40:59.384541 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:59.384510    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:40:59.384659 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:59.384596    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:40:59.384659 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:59.384643    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:40:59.814527 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:40:59.813892    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:41:00.391715 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:00.391671    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb" event={"ID":"f3bd5875-a930-4985-b820-efe272d30763","Type":"ContainerStarted","Data":"b39cb24f1a11cf12dbcbeb617d8829f681c5cc73555ea3235ead2cc4a9021764"}
Oct 30 00:41:01.249667 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:01.249637    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:41:01.250803 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:01.250767    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-cngvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 00:41:01.250865 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:01.250830    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb" podUID="f3bd5875-a930-4985-b820-efe272d30763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 00:41:02.249575 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:02.249542    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-cngvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 00:41:02.250003 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:02.249596    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb" podUID="f3bd5875-a930-4985-b820-efe272d30763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 00:41:03.249263 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:03.249224    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-cngvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 00:41:03.249462 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:03.249290    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb" podUID="f3bd5875-a930-4985-b820-efe272d30763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 00:41:04.249501 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:04.249462    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-cngvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 00:41:04.249934 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:04.249523    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb" podUID="f3bd5875-a930-4985-b820-efe272d30763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 00:41:05.249945 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:05.249904    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-cngvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 00:41:05.250438 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:05.249971    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb" podUID="f3bd5875-a930-4985-b820-efe272d30763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 00:41:06.253030 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:06.253002    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:41:06.253507 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:06.253296    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:41:06.254004 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:06.253983    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb"
Oct 30 00:41:26.510140 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:41:26.510103    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:41:50.300005 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:50.299968    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2"]
Oct 30 00:41:50.304251 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:50.304224    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2"
Oct 30 00:41:50.321417 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:50.321390    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 00:41:50.330225 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:50.330200    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 00:41:50.330410 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:50.330231    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dkdpm\""
Oct 30 00:41:50.330486 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:50.330294    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 30 00:41:50.332425 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:50.332405    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2"]
Oct 30 00:41:50.432265 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:50.432230    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kwdm\" (UniqueName: \"kubernetes.io/projected/68d0836f-44b4-45d7-a37f-bd20ce93a93f-kube-api-access-8kwdm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2\" (UID: \"68d0836f-44b4-45d7-a37f-bd20ce93a93f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2"
Oct 30 00:41:50.432435 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:50.432275    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68d0836f-44b4-45d7-a37f-bd20ce93a93f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2\" (UID: \"68d0836f-44b4-45d7-a37f-bd20ce93a93f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2"
Oct 30 00:41:50.432435 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:50.432374    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68d0836f-44b4-45d7-a37f-bd20ce93a93f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2\" (UID: \"68d0836f-44b4-45d7-a37f-bd20ce93a93f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2"
Oct 30 00:41:50.432435 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:50.432415    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68d0836f-44b4-45d7-a37f-bd20ce93a93f-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2\" (UID: \"68d0836f-44b4-45d7-a37f-bd20ce93a93f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2"
Oct 30 00:41:50.432558 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:50.432441    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68d0836f-44b4-45d7-a37f-bd20ce93a93f-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2\" (UID: \"68d0836f-44b4-45d7-a37f-bd20ce93a93f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2"
Oct 30 00:41:50.432558 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:50.432501    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68d0836f-44b4-45d7-a37f-bd20ce93a93f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2\" (UID: \"68d0836f-44b4-45d7-a37f-bd20ce93a93f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2"
Oct 30 00:41:50.533072 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:50.533019    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kwdm\" (UniqueName: \"kubernetes.io/projected/68d0836f-44b4-45d7-a37f-bd20ce93a93f-kube-api-access-8kwdm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2\" (UID: \"68d0836f-44b4-45d7-a37f-bd20ce93a93f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2"
Oct 30 00:41:50.533072 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:50.533070    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68d0836f-44b4-45d7-a37f-bd20ce93a93f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2\" (UID: \"68d0836f-44b4-45d7-a37f-bd20ce93a93f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2"
Oct 30 00:41:50.533326 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:50.533123    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68d0836f-44b4-45d7-a37f-bd20ce93a93f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2\" (UID: \"68d0836f-44b4-45d7-a37f-bd20ce93a93f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2"
Oct 30 00:41:50.533326 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:50.533156    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68d0836f-44b4-45d7-a37f-bd20ce93a93f-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2\" (UID: \"68d0836f-44b4-45d7-a37f-bd20ce93a93f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2"
Oct 30 00:41:50.533326 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:50.533181    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68d0836f-44b4-45d7-a37f-bd20ce93a93f-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2\" (UID: \"68d0836f-44b4-45d7-a37f-bd20ce93a93f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2"
Oct 30 00:41:50.533326 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:50.533218    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68d0836f-44b4-45d7-a37f-bd20ce93a93f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2\" (UID: \"68d0836f-44b4-45d7-a37f-bd20ce93a93f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2"
Oct 30 00:41:50.533795 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:50.533752    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68d0836f-44b4-45d7-a37f-bd20ce93a93f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2\" (UID: \"68d0836f-44b4-45d7-a37f-bd20ce93a93f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2"
Oct 30 00:41:50.533995 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:50.533820    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68d0836f-44b4-45d7-a37f-bd20ce93a93f-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2\" (UID: \"68d0836f-44b4-45d7-a37f-bd20ce93a93f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2"
Oct 30 00:41:50.534135 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:50.534117    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68d0836f-44b4-45d7-a37f-bd20ce93a93f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2\" (UID: \"68d0836f-44b4-45d7-a37f-bd20ce93a93f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2"
Oct 30 00:41:50.535835 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:50.535812    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68d0836f-44b4-45d7-a37f-bd20ce93a93f-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2\" (UID: \"68d0836f-44b4-45d7-a37f-bd20ce93a93f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2"
Oct 30 00:41:50.536109 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:50.536089    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68d0836f-44b4-45d7-a37f-bd20ce93a93f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2\" (UID: \"68d0836f-44b4-45d7-a37f-bd20ce93a93f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2"
Oct 30 00:41:50.555682 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:50.555621    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kwdm\" (UniqueName: \"kubernetes.io/projected/68d0836f-44b4-45d7-a37f-bd20ce93a93f-kube-api-access-8kwdm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2\" (UID: \"68d0836f-44b4-45d7-a37f-bd20ce93a93f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2"
Oct 30 00:41:50.617887 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:50.617856    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2"
Oct 30 00:41:50.917346 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:50.917322    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2"]
Oct 30 00:41:50.919106 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:41:50.919066    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d0836f_44b4_45d7_a37f_bd20ce93a93f.slice/crio-f169f5605eb3e6cb1b6466b5b521ae54ef8ec3bcc86ad58af5019deaecbf505f WatchSource:0}: Error finding container f169f5605eb3e6cb1b6466b5b521ae54ef8ec3bcc86ad58af5019deaecbf505f: Status 404 returned error can't find the container with id f169f5605eb3e6cb1b6466b5b521ae54ef8ec3bcc86ad58af5019deaecbf505f
Oct 30 00:41:51.671603 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:51.671571    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2" event={"ID":"68d0836f-44b4-45d7-a37f-bd20ce93a93f","Type":"ContainerStarted","Data":"f169f5605eb3e6cb1b6466b5b521ae54ef8ec3bcc86ad58af5019deaecbf505f"}
Oct 30 00:41:52.679086 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:52.679052    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2" event={"ID":"68d0836f-44b4-45d7-a37f-bd20ce93a93f","Type":"ContainerStarted","Data":"9c42d7fff9509c216ee81a1066dfb21ab08d37efdf3dff4be95fec701caf3aaf"}
Oct 30 00:41:52.720336 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:52.720277    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2" podStartSLOduration=1.5680118699999999 podStartE2EDuration="2.720262574s" totalImagesPullingTime="1.152250704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:41:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:52.719166517 +0000 UTC m=+603.106842907" watchObservedRunningTime="2026-10-30 00:41:52.720262574 +0000 UTC m=+603.107939019"
Oct 30 00:41:53.618741 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:53.618700    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2"
Oct 30 00:41:54.048088 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:54.047995    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c"]
Oct 30 00:41:54.054123 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:54.054094    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c"
Oct 30 00:41:54.071341 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:54.071309    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 30 00:41:54.073606 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:54.073583    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c"]
Oct 30 00:41:54.171747 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:54.171712    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkrv7\" (UniqueName: \"kubernetes.io/projected/3ff1c407-8b92-4657-82ac-eb65af7d9bbe-kube-api-access-fkrv7\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c\" (UID: \"3ff1c407-8b92-4657-82ac-eb65af7d9bbe\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c"
Oct 30 00:41:54.171747 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:54.171751    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ff1c407-8b92-4657-82ac-eb65af7d9bbe-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c\" (UID: \"3ff1c407-8b92-4657-82ac-eb65af7d9bbe\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c"
Oct 30 00:41:54.171963 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:54.171773    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ff1c407-8b92-4657-82ac-eb65af7d9bbe-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c\" (UID: \"3ff1c407-8b92-4657-82ac-eb65af7d9bbe\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c"
Oct 30 00:41:54.171963 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:54.171851    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ff1c407-8b92-4657-82ac-eb65af7d9bbe-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c\" (UID: \"3ff1c407-8b92-4657-82ac-eb65af7d9bbe\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c"
Oct 30 00:41:54.171963 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:54.171943    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ff1c407-8b92-4657-82ac-eb65af7d9bbe-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c\" (UID: \"3ff1c407-8b92-4657-82ac-eb65af7d9bbe\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c"
Oct 30 00:41:54.172075 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:54.171982    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ff1c407-8b92-4657-82ac-eb65af7d9bbe-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c\" (UID: \"3ff1c407-8b92-4657-82ac-eb65af7d9bbe\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c"
Oct 30 00:41:54.272717 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:54.272686    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ff1c407-8b92-4657-82ac-eb65af7d9bbe-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c\" (UID: \"3ff1c407-8b92-4657-82ac-eb65af7d9bbe\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c"
Oct 30 00:41:54.272903 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:54.272732    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ff1c407-8b92-4657-82ac-eb65af7d9bbe-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c\" (UID: \"3ff1c407-8b92-4657-82ac-eb65af7d9bbe\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c"
Oct 30 00:41:54.272903 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:54.272866    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkrv7\" (UniqueName: \"kubernetes.io/projected/3ff1c407-8b92-4657-82ac-eb65af7d9bbe-kube-api-access-fkrv7\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c\" (UID: \"3ff1c407-8b92-4657-82ac-eb65af7d9bbe\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c"
Oct 30 00:41:54.273009 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:54.272906    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ff1c407-8b92-4657-82ac-eb65af7d9bbe-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c\" (UID: \"3ff1c407-8b92-4657-82ac-eb65af7d9bbe\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c"
Oct 30 00:41:54.273009 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:54.272934    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ff1c407-8b92-4657-82ac-eb65af7d9bbe-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c\" (UID: \"3ff1c407-8b92-4657-82ac-eb65af7d9bbe\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c"
Oct 30 00:41:54.273009 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:54.272949    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ff1c407-8b92-4657-82ac-eb65af7d9bbe-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c\" (UID: \"3ff1c407-8b92-4657-82ac-eb65af7d9bbe\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c"
Oct 30 00:41:54.273201 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:54.273182    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ff1c407-8b92-4657-82ac-eb65af7d9bbe-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c\" (UID: \"3ff1c407-8b92-4657-82ac-eb65af7d9bbe\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c"
Oct 30 00:41:54.273290 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:54.273242    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ff1c407-8b92-4657-82ac-eb65af7d9bbe-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c\" (UID: \"3ff1c407-8b92-4657-82ac-eb65af7d9bbe\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c"
Oct 30 00:41:54.273353 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:54.273295    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ff1c407-8b92-4657-82ac-eb65af7d9bbe-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c\" (UID: \"3ff1c407-8b92-4657-82ac-eb65af7d9bbe\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c"
Oct 30 00:41:54.275256 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:54.275235    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ff1c407-8b92-4657-82ac-eb65af7d9bbe-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c\" (UID: \"3ff1c407-8b92-4657-82ac-eb65af7d9bbe\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c"
Oct 30 00:41:54.275599 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:54.275578    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ff1c407-8b92-4657-82ac-eb65af7d9bbe-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c\" (UID: \"3ff1c407-8b92-4657-82ac-eb65af7d9bbe\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c"
Oct 30 00:41:54.295399 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:54.295377    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkrv7\" (UniqueName: \"kubernetes.io/projected/3ff1c407-8b92-4657-82ac-eb65af7d9bbe-kube-api-access-fkrv7\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c\" (UID: \"3ff1c407-8b92-4657-82ac-eb65af7d9bbe\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c"
Oct 30 00:41:54.369480 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:54.369456    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c"
Oct 30 00:41:54.667826 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:54.667797    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c"]
Oct 30 00:41:54.669386 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:41:54.669364    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff1c407_8b92_4657_82ac_eb65af7d9bbe.slice/crio-a9b12b6b512580ad2cf065d1d6a745d9493401b983f3b2ca8898f3322a4da5b2 WatchSource:0}: Error finding container a9b12b6b512580ad2cf065d1d6a745d9493401b983f3b2ca8898f3322a4da5b2: Status 404 returned error can't find the container with id a9b12b6b512580ad2cf065d1d6a745d9493401b983f3b2ca8898f3322a4da5b2
Oct 30 00:41:54.690834 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:54.690805    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c" event={"ID":"3ff1c407-8b92-4657-82ac-eb65af7d9bbe","Type":"ContainerStarted","Data":"a9b12b6b512580ad2cf065d1d6a745d9493401b983f3b2ca8898f3322a4da5b2"}
Oct 30 00:41:55.698408 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:55.698373    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c" event={"ID":"3ff1c407-8b92-4657-82ac-eb65af7d9bbe","Type":"ContainerStarted","Data":"ebab01f7da6f3168d9967d71988a1e22a216a42956b63301de91cbc063babc3a"}
Oct 30 00:41:55.743702 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:55.743653    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c" podStartSLOduration=1.563545001 podStartE2EDuration="1.743640349s" totalImagesPullingTime="180.095348ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:41:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:41:55.74181522 +0000 UTC m=+606.129491608" watchObservedRunningTime="2026-10-30 00:41:55.743640349 +0000 UTC m=+606.131316737"
Oct 30 00:41:56.370448 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:41:56.370416    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c"
Oct 30 00:41:56.514077 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:41:56.514035    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:42:00.001260 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:00.001219    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-s7t4h"]
Oct 30 00:42:00.005434 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:00.005401    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s7t4h"
Oct 30 00:42:00.036563 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:00.035990    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-s7t4h"]
Oct 30 00:42:00.043270 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:00.043232    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 30 00:42:00.128237 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:00.128186    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/770ca287-b172-4664-9e09-f024674f5c9c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s7t4h\" (UID: \"770ca287-b172-4664-9e09-f024674f5c9c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s7t4h"
Oct 30 00:42:00.128237 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:00.128232    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/770ca287-b172-4664-9e09-f024674f5c9c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s7t4h\" (UID: \"770ca287-b172-4664-9e09-f024674f5c9c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s7t4h"
Oct 30 00:42:00.128508 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:00.128264    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/770ca287-b172-4664-9e09-f024674f5c9c-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s7t4h\" (UID: \"770ca287-b172-4664-9e09-f024674f5c9c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s7t4h"
Oct 30 00:42:00.128508 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:00.128401    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/770ca287-b172-4664-9e09-f024674f5c9c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s7t4h\" (UID: \"770ca287-b172-4664-9e09-f024674f5c9c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s7t4h"
Oct 30 00:42:00.128508 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:00.128439    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/770ca287-b172-4664-9e09-f024674f5c9c-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s7t4h\" (UID: \"770ca287-b172-4664-9e09-f024674f5c9c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s7t4h"
Oct 30 00:42:00.128682 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:00.128521    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kt6m\" (UniqueName: \"kubernetes.io/projected/770ca287-b172-4664-9e09-f024674f5c9c-kube-api-access-8kt6m\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s7t4h\" (UID: \"770ca287-b172-4664-9e09-f024674f5c9c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s7t4h"
Oct 30 00:42:00.229401 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:00.229367    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kt6m\" (UniqueName: \"kubernetes.io/projected/770ca287-b172-4664-9e09-f024674f5c9c-kube-api-access-8kt6m\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s7t4h\" (UID: \"770ca287-b172-4664-9e09-f024674f5c9c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s7t4h"
Oct 30 00:42:00.229609 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:00.229440    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/770ca287-b172-4664-9e09-f024674f5c9c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s7t4h\" (UID: \"770ca287-b172-4664-9e09-f024674f5c9c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s7t4h"
Oct 30 00:42:00.229609 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:00.229485    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/770ca287-b172-4664-9e09-f024674f5c9c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s7t4h\" (UID: \"770ca287-b172-4664-9e09-f024674f5c9c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s7t4h"
Oct 30 00:42:00.229609 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:00.229519    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/770ca287-b172-4664-9e09-f024674f5c9c-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s7t4h\" (UID: \"770ca287-b172-4664-9e09-f024674f5c9c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s7t4h"
Oct 30 00:42:00.229609 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:00.229549    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/770ca287-b172-4664-9e09-f024674f5c9c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s7t4h\" (UID: \"770ca287-b172-4664-9e09-f024674f5c9c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s7t4h"
Oct 30 00:42:00.229911 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:00.229887    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/770ca287-b172-4664-9e09-f024674f5c9c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s7t4h\" (UID: \"770ca287-b172-4664-9e09-f024674f5c9c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s7t4h"
Oct 30 00:42:00.229911 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:00.229904    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/770ca287-b172-4664-9e09-f024674f5c9c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s7t4h\" (UID: \"770ca287-b172-4664-9e09-f024674f5c9c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s7t4h"
Oct 30 00:42:00.230031 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:00.229942    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/770ca287-b172-4664-9e09-f024674f5c9c-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s7t4h\" (UID: \"770ca287-b172-4664-9e09-f024674f5c9c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s7t4h"
Oct 30 00:42:00.230235 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:00.230205    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/770ca287-b172-4664-9e09-f024674f5c9c-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s7t4h\" (UID: \"770ca287-b172-4664-9e09-f024674f5c9c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s7t4h"
Oct 30 00:42:00.232527 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:00.232493    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/770ca287-b172-4664-9e09-f024674f5c9c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s7t4h\" (UID: \"770ca287-b172-4664-9e09-f024674f5c9c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s7t4h"
Oct 30 00:42:00.232656 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:00.232494    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/770ca287-b172-4664-9e09-f024674f5c9c-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s7t4h\" (UID: \"770ca287-b172-4664-9e09-f024674f5c9c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s7t4h"
Oct 30 00:42:00.253414 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:00.253337    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kt6m\" (UniqueName: \"kubernetes.io/projected/770ca287-b172-4664-9e09-f024674f5c9c-kube-api-access-8kt6m\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-s7t4h\" (UID: \"770ca287-b172-4664-9e09-f024674f5c9c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s7t4h"
Oct 30 00:42:00.343176 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:00.343133    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s7t4h"
Oct 30 00:42:00.618478 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:00.618453    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2"
Oct 30 00:42:00.631502 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:00.631474    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2"
Oct 30 00:42:00.652333 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:00.652308    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-s7t4h"]
Oct 30 00:42:00.653737 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:42:00.653714    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod770ca287_b172_4664_9e09_f024674f5c9c.slice/crio-f9600881ce5d9af5111cd0073f7aa2a27a6e76342f04bcc0fa3ee260298a619a WatchSource:0}: Error finding container f9600881ce5d9af5111cd0073f7aa2a27a6e76342f04bcc0fa3ee260298a619a: Status 404 returned error can't find the container with id f9600881ce5d9af5111cd0073f7aa2a27a6e76342f04bcc0fa3ee260298a619a
Oct 30 00:42:00.724839 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:00.724805    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s7t4h" event={"ID":"770ca287-b172-4664-9e09-f024674f5c9c","Type":"ContainerStarted","Data":"f9600881ce5d9af5111cd0073f7aa2a27a6e76342f04bcc0fa3ee260298a619a"}
Oct 30 00:42:00.736262 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:00.736233    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2"
Oct 30 00:42:01.731809 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:01.731751    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s7t4h" event={"ID":"770ca287-b172-4664-9e09-f024674f5c9c","Type":"ContainerStarted","Data":"e5abf5f298fd86f351978ddeb7a6d099d77621bee6814c723ded0bc483b7037a"}
Oct 30 00:42:01.772718 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:01.772673    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s7t4h" podStartSLOduration=2.598018652 podStartE2EDuration="2.772660119s" totalImagesPullingTime="174.641467ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:41:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:01.771945156 +0000 UTC m=+612.159621544" watchObservedRunningTime="2026-10-30 00:42:01.772660119 +0000 UTC m=+612.160336508"
Oct 30 00:42:02.344290 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:02.344259    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s7t4h"
Oct 30 00:42:04.369843 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:04.369808    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c"
Oct 30 00:42:04.382432 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:04.382402    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c"
Oct 30 00:42:04.758941 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:04.758859    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c"
Oct 30 00:42:06.414897 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:06.414850    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-d8c8ccb-dpdv8"]
Oct 30 00:42:06.418281 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:06.418261    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-d8c8ccb-dpdv8"
Oct 30 00:42:06.437737 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:06.437714    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-d8c8ccb-dpdv8"]
Oct 30 00:42:06.446990 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:06.446880    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-xsvpd\""
Oct 30 00:42:06.446990 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:06.446942    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 00:42:06.446990 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:06.446968    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 30 00:42:06.492195 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:06.492161    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96285e7a-0516-414a-bfc5-295a89aaf62f-metrics-certs\") pod \"ai-gateway-controller-d8c8ccb-dpdv8\" (UID: \"96285e7a-0516-414a-bfc5-295a89aaf62f\") " pod="opendatahub/ai-gateway-controller-d8c8ccb-dpdv8"
Oct 30 00:42:06.492363 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:06.492285    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f7gz\" (UniqueName: \"kubernetes.io/projected/96285e7a-0516-414a-bfc5-295a89aaf62f-kube-api-access-5f7gz\") pod \"ai-gateway-controller-d8c8ccb-dpdv8\" (UID: \"96285e7a-0516-414a-bfc5-295a89aaf62f\") " pod="opendatahub/ai-gateway-controller-d8c8ccb-dpdv8"
Oct 30 00:42:06.593392 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:06.593356    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f7gz\" (UniqueName: \"kubernetes.io/projected/96285e7a-0516-414a-bfc5-295a89aaf62f-kube-api-access-5f7gz\") pod \"ai-gateway-controller-d8c8ccb-dpdv8\" (UID: \"96285e7a-0516-414a-bfc5-295a89aaf62f\") " pod="opendatahub/ai-gateway-controller-d8c8ccb-dpdv8"
Oct 30 00:42:06.593547 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:06.593450    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96285e7a-0516-414a-bfc5-295a89aaf62f-metrics-certs\") pod \"ai-gateway-controller-d8c8ccb-dpdv8\" (UID: \"96285e7a-0516-414a-bfc5-295a89aaf62f\") " pod="opendatahub/ai-gateway-controller-d8c8ccb-dpdv8"
Oct 30 00:42:06.596029 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:06.596006    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96285e7a-0516-414a-bfc5-295a89aaf62f-metrics-certs\") pod \"ai-gateway-controller-d8c8ccb-dpdv8\" (UID: \"96285e7a-0516-414a-bfc5-295a89aaf62f\") " pod="opendatahub/ai-gateway-controller-d8c8ccb-dpdv8"
Oct 30 00:42:06.615635 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:06.615606    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f7gz\" (UniqueName: \"kubernetes.io/projected/96285e7a-0516-414a-bfc5-295a89aaf62f-kube-api-access-5f7gz\") pod \"ai-gateway-controller-d8c8ccb-dpdv8\" (UID: \"96285e7a-0516-414a-bfc5-295a89aaf62f\") " pod="opendatahub/ai-gateway-controller-d8c8ccb-dpdv8"
Oct 30 00:42:06.731566 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:06.731467    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-d8c8ccb-dpdv8"
Oct 30 00:42:07.030573 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:07.030547    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-d8c8ccb-dpdv8"]
Oct 30 00:42:07.032079 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:42:07.032049    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96285e7a_0516_414a_bfc5_295a89aaf62f.slice/crio-19d2eb1d1fd2d976dede24e3f78b196701242b6e66fd288481d1776faec1a159 WatchSource:0}: Error finding container 19d2eb1d1fd2d976dede24e3f78b196701242b6e66fd288481d1776faec1a159: Status 404 returned error can't find the container with id 19d2eb1d1fd2d976dede24e3f78b196701242b6e66fd288481d1776faec1a159
Oct 30 00:42:07.766217 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:07.766180    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-d8c8ccb-dpdv8" event={"ID":"96285e7a-0516-414a-bfc5-295a89aaf62f","Type":"ContainerStarted","Data":"19d2eb1d1fd2d976dede24e3f78b196701242b6e66fd288481d1776faec1a159"}
Oct 30 00:42:09.777808 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:09.777751    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-d8c8ccb-dpdv8" event={"ID":"96285e7a-0516-414a-bfc5-295a89aaf62f","Type":"ContainerStarted","Data":"40b033a47158174bbfe0718df602ea616b89a432b14827cbb2378950b8dd36a3"}
Oct 30 00:42:09.778201 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:09.777914    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-d8c8ccb-dpdv8"
Oct 30 00:42:09.817805 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:09.817717    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-d8c8ccb-dpdv8" podStartSLOduration=1.550157869 podStartE2EDuration="3.817703683s" totalImagesPullingTime="2.267545814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:09.816018303 +0000 UTC m=+620.203694691" watchObservedRunningTime="2026-10-30 00:42:09.817703683 +0000 UTC m=+620.205380071"
Oct 30 00:42:10.343731 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:10.343693    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s7t4h"
Oct 30 00:42:10.356526 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:10.356506    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s7t4h"
Oct 30 00:42:10.794646 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:10.794613    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-s7t4h"
Oct 30 00:42:11.674254 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:11.674223    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-l4gst"]
Oct 30 00:42:11.678122 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:11.678101    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-l4gst"
Oct 30 00:42:11.694359 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:11.694332    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 30 00:42:11.694547 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:11.694466    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-9hmph\""
Oct 30 00:42:11.697431 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:11.697411    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-l4gst"]
Oct 30 00:42:11.746958 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:11.746926    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnvdt\" (UniqueName: \"kubernetes.io/projected/9217cef0-6bf2-41d5-b303-4c4b78dcb7f0-kube-api-access-pnvdt\") pod \"payload-processing-5f78687d5b-l4gst\" (UID: \"9217cef0-6bf2-41d5-b303-4c4b78dcb7f0\") " pod="openshift-ingress/payload-processing-5f78687d5b-l4gst"
Oct 30 00:42:11.747124 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:11.747050    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9217cef0-6bf2-41d5-b303-4c4b78dcb7f0-plugins-config-volume\") pod \"payload-processing-5f78687d5b-l4gst\" (UID: \"9217cef0-6bf2-41d5-b303-4c4b78dcb7f0\") " pod="openshift-ingress/payload-processing-5f78687d5b-l4gst"
Oct 30 00:42:11.848068 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:11.848038    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnvdt\" (UniqueName: \"kubernetes.io/projected/9217cef0-6bf2-41d5-b303-4c4b78dcb7f0-kube-api-access-pnvdt\") pod \"payload-processing-5f78687d5b-l4gst\" (UID: \"9217cef0-6bf2-41d5-b303-4c4b78dcb7f0\") " pod="openshift-ingress/payload-processing-5f78687d5b-l4gst"
Oct 30 00:42:11.848521 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:11.848119    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9217cef0-6bf2-41d5-b303-4c4b78dcb7f0-plugins-config-volume\") pod \"payload-processing-5f78687d5b-l4gst\" (UID: \"9217cef0-6bf2-41d5-b303-4c4b78dcb7f0\") " pod="openshift-ingress/payload-processing-5f78687d5b-l4gst"
Oct 30 00:42:11.849022 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:11.848996    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9217cef0-6bf2-41d5-b303-4c4b78dcb7f0-plugins-config-volume\") pod \"payload-processing-5f78687d5b-l4gst\" (UID: \"9217cef0-6bf2-41d5-b303-4c4b78dcb7f0\") " pod="openshift-ingress/payload-processing-5f78687d5b-l4gst"
Oct 30 00:42:11.871009 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:11.870982    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnvdt\" (UniqueName: \"kubernetes.io/projected/9217cef0-6bf2-41d5-b303-4c4b78dcb7f0-kube-api-access-pnvdt\") pod \"payload-processing-5f78687d5b-l4gst\" (UID: \"9217cef0-6bf2-41d5-b303-4c4b78dcb7f0\") " pod="openshift-ingress/payload-processing-5f78687d5b-l4gst"
Oct 30 00:42:11.991888 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:11.991805    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-l4gst"
Oct 30 00:42:12.288118 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:12.288094    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-l4gst"]
Oct 30 00:42:12.289615 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:42:12.289588    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9217cef0_6bf2_41d5_b303_4c4b78dcb7f0.slice/crio-54a9d2c88dfbbaa11645f4950a3e480d297d6827e1a4e3bd2d2c4255561bda50 WatchSource:0}: Error finding container 54a9d2c88dfbbaa11645f4950a3e480d297d6827e1a4e3bd2d2c4255561bda50: Status 404 returned error can't find the container with id 54a9d2c88dfbbaa11645f4950a3e480d297d6827e1a4e3bd2d2c4255561bda50
Oct 30 00:42:12.799078 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:12.799040    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-l4gst" event={"ID":"9217cef0-6bf2-41d5-b303-4c4b78dcb7f0","Type":"ContainerStarted","Data":"54a9d2c88dfbbaa11645f4950a3e480d297d6827e1a4e3bd2d2c4255561bda50"}
Oct 30 00:42:14.812378 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:14.812339    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-l4gst" event={"ID":"9217cef0-6bf2-41d5-b303-4c4b78dcb7f0","Type":"ContainerStarted","Data":"93cadd09e184c9525d59f9daa56406e11736d099083e2a5fcc8e2104ecd80070"}
Oct 30 00:42:14.812810 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:14.812417    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-l4gst"
Oct 30 00:42:14.851360 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:14.851313    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-l4gst" podStartSLOduration=1.846292322 podStartE2EDuration="3.851301406s" totalImagesPullingTime="2.005009084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:14.850414045 +0000 UTC m=+625.238090434" watchObservedRunningTime="2026-10-30 00:42:14.851301406 +0000 UTC m=+625.238977794"
Oct 30 00:42:20.785618 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:20.785589    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-d8c8ccb-dpdv8"
Oct 30 00:42:21.065090 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:21.065008    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7fd968db48-lmzwp"]
Oct 30 00:42:21.068920 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:21.068899    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7fd968db48-lmzwp"
Oct 30 00:42:21.088980 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:21.088955    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7fd968db48-lmzwp"]
Oct 30 00:42:21.142533 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:21.142503    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95z8q\" (UniqueName: \"kubernetes.io/projected/afe02d9e-6306-48f3-adde-1e5aa9381b83-kube-api-access-95z8q\") pod \"ai-gateway-controller-7fd968db48-lmzwp\" (UID: \"afe02d9e-6306-48f3-adde-1e5aa9381b83\") " pod="opendatahub/ai-gateway-controller-7fd968db48-lmzwp"
Oct 30 00:42:21.142723 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:21.142542    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afe02d9e-6306-48f3-adde-1e5aa9381b83-metrics-certs\") pod \"ai-gateway-controller-7fd968db48-lmzwp\" (UID: \"afe02d9e-6306-48f3-adde-1e5aa9381b83\") " pod="opendatahub/ai-gateway-controller-7fd968db48-lmzwp"
Oct 30 00:42:21.243186 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:21.243158    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95z8q\" (UniqueName: \"kubernetes.io/projected/afe02d9e-6306-48f3-adde-1e5aa9381b83-kube-api-access-95z8q\") pod \"ai-gateway-controller-7fd968db48-lmzwp\" (UID: \"afe02d9e-6306-48f3-adde-1e5aa9381b83\") " pod="opendatahub/ai-gateway-controller-7fd968db48-lmzwp"
Oct 30 00:42:21.243377 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:21.243203    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afe02d9e-6306-48f3-adde-1e5aa9381b83-metrics-certs\") pod \"ai-gateway-controller-7fd968db48-lmzwp\" (UID: \"afe02d9e-6306-48f3-adde-1e5aa9381b83\") " pod="opendatahub/ai-gateway-controller-7fd968db48-lmzwp"
Oct 30 00:42:21.245725 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:21.245699    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afe02d9e-6306-48f3-adde-1e5aa9381b83-metrics-certs\") pod \"ai-gateway-controller-7fd968db48-lmzwp\" (UID: \"afe02d9e-6306-48f3-adde-1e5aa9381b83\") " pod="opendatahub/ai-gateway-controller-7fd968db48-lmzwp"
Oct 30 00:42:21.266593 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:21.266568    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95z8q\" (UniqueName: \"kubernetes.io/projected/afe02d9e-6306-48f3-adde-1e5aa9381b83-kube-api-access-95z8q\") pod \"ai-gateway-controller-7fd968db48-lmzwp\" (UID: \"afe02d9e-6306-48f3-adde-1e5aa9381b83\") " pod="opendatahub/ai-gateway-controller-7fd968db48-lmzwp"
Oct 30 00:42:21.381723 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:21.381697    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7fd968db48-lmzwp"
Oct 30 00:42:21.681338 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:21.681304    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7fd968db48-lmzwp"]
Oct 30 00:42:21.683200 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:42:21.683169    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe02d9e_6306_48f3_adde_1e5aa9381b83.slice/crio-40dface402813ab79cb816c70e006ece0ff24a2d6b1b232ebccf5a01caaff903 WatchSource:0}: Error finding container 40dface402813ab79cb816c70e006ece0ff24a2d6b1b232ebccf5a01caaff903: Status 404 returned error can't find the container with id 40dface402813ab79cb816c70e006ece0ff24a2d6b1b232ebccf5a01caaff903
Oct 30 00:42:21.850221 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:21.850186    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7fd968db48-lmzwp" event={"ID":"afe02d9e-6306-48f3-adde-1e5aa9381b83","Type":"ContainerStarted","Data":"40dface402813ab79cb816c70e006ece0ff24a2d6b1b232ebccf5a01caaff903"}
Oct 30 00:42:22.856796 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:22.856751    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7fd968db48-lmzwp" event={"ID":"afe02d9e-6306-48f3-adde-1e5aa9381b83","Type":"ContainerStarted","Data":"5969ba22278725b6bcfb1f9c4db2ffb1d06c62916bdd9d9c824a99dd55d78d4f"}
Oct 30 00:42:22.857302 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:22.856917    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7fd968db48-lmzwp"
Oct 30 00:42:22.897405 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:22.897349    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7fd968db48-lmzwp" podStartSLOduration=1.584655645 podStartE2EDuration="1.897336741s" totalImagesPullingTime="312.681096ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:22.89580364 +0000 UTC m=+633.283480024" watchObservedRunningTime="2026-10-30 00:42:22.897336741 +0000 UTC m=+633.285013130"
Oct 30 00:42:25.821048 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:25.821018    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-l4gst"
Oct 30 00:42:26.509531 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:42:26.509488    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:42:29.312086 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:29.312052    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"]
Oct 30 00:42:29.315789 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:29.315759    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"
Oct 30 00:42:29.346170 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:29.346139    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"]
Oct 30 00:42:29.421174 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:29.421144    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xx8n6\" (UID: \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"
Oct 30 00:42:29.421344 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:29.421198    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xx8n6\" (UID: \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"
Oct 30 00:42:29.421344 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:29.421216    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xx8n6\" (UID: \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"
Oct 30 00:42:29.421417 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:29.421337    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xx8n6\" (UID: \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"
Oct 30 00:42:29.421417 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:29.421375    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xx8n6\" (UID: \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"
Oct 30 00:42:29.421484 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:29.421437    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xx8n6\" (UID: \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"
Oct 30 00:42:29.421484 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:29.421455    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xx8n6\" (UID: \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"
Oct 30 00:42:29.421555 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:29.421542    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xx8n6\" (UID: \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"
Oct 30 00:42:29.421598 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:29.421588    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-756zh\" (UniqueName: \"kubernetes.io/projected/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-kube-api-access-756zh\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xx8n6\" (UID: \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"
Oct 30 00:42:29.522247 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:29.522213    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xx8n6\" (UID: \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"
Oct 30 00:42:29.522405 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:29.522264    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xx8n6\" (UID: \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"
Oct 30 00:42:29.522405 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:29.522287    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xx8n6\" (UID: \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"
Oct 30 00:42:29.522405 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:29.522324    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xx8n6\" (UID: \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"
Oct 30 00:42:29.522405 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:29.522362    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xx8n6\" (UID: \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"
Oct 30 00:42:29.522639 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:29.522407    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xx8n6\" (UID: \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"
Oct 30 00:42:29.522639 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:29.522430    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xx8n6\" (UID: \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"
Oct 30 00:42:29.522639 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:29.522467    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xx8n6\" (UID: \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"
Oct 30 00:42:29.522639 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:29.522522    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-756zh\" (UniqueName: \"kubernetes.io/projected/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-kube-api-access-756zh\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xx8n6\" (UID: \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"
Oct 30 00:42:29.522881 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:29.522661    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xx8n6\" (UID: \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"
Oct 30 00:42:29.522881 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:29.522760    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xx8n6\" (UID: \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"
Oct 30 00:42:29.522881 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:29.522812    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xx8n6\" (UID: \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"
Oct 30 00:42:29.523046 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:29.523028    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xx8n6\" (UID: \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"
Oct 30 00:42:29.523158 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:29.523139    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xx8n6\" (UID: \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"
Oct 30 00:42:29.524935 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:29.524914    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xx8n6\" (UID: \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"
Oct 30 00:42:29.525070 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:29.525050    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xx8n6\" (UID: \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"
Oct 30 00:42:29.545383 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:29.545352    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xx8n6\" (UID: \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"
Oct 30 00:42:29.545624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:29.545605    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-756zh\" (UniqueName: \"kubernetes.io/projected/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-kube-api-access-756zh\") pod \"maas-default-gateway-openshift-default-7d9897b54c-xx8n6\" (UID: \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"
Oct 30 00:42:29.630754 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:29.630728    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"
Oct 30 00:42:29.944449 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:29.944427    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"]
Oct 30 00:42:29.945854 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:42:29.945829    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc80bac8_5fa1_4c5e_bdb1_73225ad3181c.slice/crio-4cdc9180ae99aa0cf9684ec9484661796a64d4fdae54017deedcb0106f9587fe WatchSource:0}: Error finding container 4cdc9180ae99aa0cf9684ec9484661796a64d4fdae54017deedcb0106f9587fe: Status 404 returned error can't find the container with id 4cdc9180ae99aa0cf9684ec9484661796a64d4fdae54017deedcb0106f9587fe
Oct 30 00:42:29.947954 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:29.947920    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:42:29.948031 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:29.947986    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:42:29.948031 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:29.948017    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:42:30.900326 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:30.900296    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6" event={"ID":"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c","Type":"ContainerStarted","Data":"118235a0a3a884f51100eb415119c01c80bc67f64d03eb58c146604fcad69cab"}
Oct 30 00:42:30.900734 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:30.900343    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6" event={"ID":"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c","Type":"ContainerStarted","Data":"4cdc9180ae99aa0cf9684ec9484661796a64d4fdae54017deedcb0106f9587fe"}
Oct 30 00:42:30.961929 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:30.961876    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6" podStartSLOduration=1.96186163 podStartE2EDuration="1.96186163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:42:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:42:30.959750698 +0000 UTC m=+641.347427089" watchObservedRunningTime="2026-10-30 00:42:30.96186163 +0000 UTC m=+641.349538019"
Oct 30 00:42:31.631316 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:31.631272    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"
Oct 30 00:42:31.632911 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:31.632884    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-xx8n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 00:42:31.633036 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:31.632933    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6" podUID="dc80bac8-5fa1-4c5e-bdb1-73225ad3181c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 00:42:32.631326 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:32.631290    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-xx8n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 00:42:32.631739 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:32.631345    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6" podUID="dc80bac8-5fa1-4c5e-bdb1-73225ad3181c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 00:42:33.631327 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:33.631295    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-xx8n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 00:42:33.631501 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:33.631346    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6" podUID="dc80bac8-5fa1-4c5e-bdb1-73225ad3181c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 00:42:33.865870 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:33.865835    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7fd968db48-lmzwp"
Oct 30 00:42:33.940713 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:33.940605    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-d8c8ccb-dpdv8"]
Oct 30 00:42:33.940953 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:33.940844    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-d8c8ccb-dpdv8" podUID="96285e7a-0516-414a-bfc5-295a89aaf62f" containerName="manager" containerID="cri-o://40b033a47158174bbfe0718df602ea616b89a432b14827cbb2378950b8dd36a3" gracePeriod=30
Oct 30 00:42:34.234537 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:34.234510    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-d8c8ccb-dpdv8"
Oct 30 00:42:34.379311 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:34.379270    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96285e7a-0516-414a-bfc5-295a89aaf62f-kube-api-access-5f7gz\" (UniqueName: \"kubernetes.io/projected/96285e7a-0516-414a-bfc5-295a89aaf62f-kube-api-access-5f7gz\") pod \"96285e7a-0516-414a-bfc5-295a89aaf62f\" (UID: \"96285e7a-0516-414a-bfc5-295a89aaf62f\") "
Oct 30 00:42:34.379502 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:34.379397    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96285e7a-0516-414a-bfc5-295a89aaf62f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/96285e7a-0516-414a-bfc5-295a89aaf62f-metrics-certs\") pod \"96285e7a-0516-414a-bfc5-295a89aaf62f\" (UID: \"96285e7a-0516-414a-bfc5-295a89aaf62f\") "
Oct 30 00:42:34.382066 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:34.382037    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96285e7a-0516-414a-bfc5-295a89aaf62f-metrics-certs" pod "96285e7a-0516-414a-bfc5-295a89aaf62f" (UID: "96285e7a-0516-414a-bfc5-295a89aaf62f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:42:34.382188 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:34.382076    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96285e7a-0516-414a-bfc5-295a89aaf62f-kube-api-access-5f7gz" pod "96285e7a-0516-414a-bfc5-295a89aaf62f" (UID: "96285e7a-0516-414a-bfc5-295a89aaf62f"). InnerVolumeSpecName "kube-api-access-5f7gz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:42:34.481051 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:34.480957    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f7gz\" (UniqueName: \"kubernetes.io/projected/96285e7a-0516-414a-bfc5-295a89aaf62f-kube-api-access-5f7gz\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:42:34.481051 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:34.480992    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96285e7a-0516-414a-bfc5-295a89aaf62f-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:42:34.631364 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:34.631322    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-xx8n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 00:42:34.631542 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:34.631388    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6" podUID="dc80bac8-5fa1-4c5e-bdb1-73225ad3181c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 00:42:34.946271 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:34.946236    4393 generic.go:356] "Generic (PLEG): container finished" podID="96285e7a-0516-414a-bfc5-295a89aaf62f" containerID="40b033a47158174bbfe0718df602ea616b89a432b14827cbb2378950b8dd36a3" exitCode=0
Oct 30 00:42:34.946479 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:34.946283    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-d8c8ccb-dpdv8"
Oct 30 00:42:34.946479 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:34.946326    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-d8c8ccb-dpdv8" event={"ID":"96285e7a-0516-414a-bfc5-295a89aaf62f","Type":"ContainerDied","Data":"40b033a47158174bbfe0718df602ea616b89a432b14827cbb2378950b8dd36a3"}
Oct 30 00:42:34.946479 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:34.946383    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-d8c8ccb-dpdv8" event={"ID":"96285e7a-0516-414a-bfc5-295a89aaf62f","Type":"ContainerDied","Data":"19d2eb1d1fd2d976dede24e3f78b196701242b6e66fd288481d1776faec1a159"}
Oct 30 00:42:34.946479 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:34.946414    4393 scope.go:122] "RemoveContainer" containerID="40b033a47158174bbfe0718df602ea616b89a432b14827cbb2378950b8dd36a3"
Oct 30 00:42:34.956450 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:34.956436    4393 scope.go:122] "RemoveContainer" containerID="40b033a47158174bbfe0718df602ea616b89a432b14827cbb2378950b8dd36a3"
Oct 30 00:42:34.956695 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:42:34.956676    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40b033a47158174bbfe0718df602ea616b89a432b14827cbb2378950b8dd36a3\": container with ID starting with 40b033a47158174bbfe0718df602ea616b89a432b14827cbb2378950b8dd36a3 not found: ID does not exist" containerID="40b033a47158174bbfe0718df602ea616b89a432b14827cbb2378950b8dd36a3"
Oct 30 00:42:34.956760 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:34.956705    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40b033a47158174bbfe0718df602ea616b89a432b14827cbb2378950b8dd36a3"} err="failed to get container status \"40b033a47158174bbfe0718df602ea616b89a432b14827cbb2378950b8dd36a3\": rpc error: code = NotFound desc = could not find container \"40b033a47158174bbfe0718df602ea616b89a432b14827cbb2378950b8dd36a3\": container with ID starting with 40b033a47158174bbfe0718df602ea616b89a432b14827cbb2378950b8dd36a3 not found: ID does not exist"
Oct 30 00:42:34.995300 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:34.995271    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-d8c8ccb-dpdv8"]
Oct 30 00:42:35.027029 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:35.026998    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-d8c8ccb-dpdv8"]
Oct 30 00:42:35.631284 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:35.631251    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-xx8n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 00:42:35.631461 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:35.631332    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6" podUID="dc80bac8-5fa1-4c5e-bdb1-73225ad3181c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 00:42:35.731261 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:35.731222    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96285e7a-0516-414a-bfc5-295a89aaf62f" path="/var/lib/kubelet/pods/96285e7a-0516-414a-bfc5-295a89aaf62f/volumes"
Oct 30 00:42:36.631988 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:36.631950    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-xx8n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 00:42:36.632173 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:36.632013    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6" podUID="dc80bac8-5fa1-4c5e-bdb1-73225ad3181c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 00:42:38.185105 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:38.185067    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"
Oct 30 00:42:38.185655 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:38.185625    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"
Oct 30 00:42:38.186393 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:42:38.186373    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"
Oct 30 00:42:56.509723 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:42:56.509684    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:43:05.145917 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.145825    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"]
Oct 30 00:43:05.146537 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.146518    4393 cpu_manager.go:414] "containerMap: removing container" podUID="96285e7a-0516-414a-bfc5-295a89aaf62f" containerName="manager"
Oct 30 00:43:05.146600 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.146539    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96285e7a-0516-414a-bfc5-295a89aaf62f" containerName="manager"
Oct 30 00:43:05.146697 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.146684    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="96285e7a-0516-414a-bfc5-295a89aaf62f" containerName="manager"
Oct 30 00:43:05.149463 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.149443    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:05.163979 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.163956    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-52650f-openshift-default-dockercfg-5tktc\""
Oct 30 00:43:05.179425 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.179393    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"]
Oct 30 00:43:05.272943 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.272908    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4370a67-835f-4d52-97c4-0419593f7975-istio-podinfo\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x\" (UID: \"d4370a67-835f-4d52-97c4-0419593f7975\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:05.272943 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.272945    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4370a67-835f-4d52-97c4-0419593f7975-istio-data\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x\" (UID: \"d4370a67-835f-4d52-97c4-0419593f7975\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:05.273182 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.272978    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4370a67-835f-4d52-97c4-0419593f7975-istio-envoy\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x\" (UID: \"d4370a67-835f-4d52-97c4-0419593f7975\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:05.273182 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.273091    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4370a67-835f-4d52-97c4-0419593f7975-credential-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x\" (UID: \"d4370a67-835f-4d52-97c4-0419593f7975\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:05.273182 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.273130    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4370a67-835f-4d52-97c4-0419593f7975-istiod-ca-cert\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x\" (UID: \"d4370a67-835f-4d52-97c4-0419593f7975\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:05.273182 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.273152    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4370a67-835f-4d52-97c4-0419593f7975-workload-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x\" (UID: \"d4370a67-835f-4d52-97c4-0419593f7975\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:05.273355 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.273226    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2w6b\" (UniqueName: \"kubernetes.io/projected/d4370a67-835f-4d52-97c4-0419593f7975-kube-api-access-l2w6b\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x\" (UID: \"d4370a67-835f-4d52-97c4-0419593f7975\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:05.273355 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.273258    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4370a67-835f-4d52-97c4-0419593f7975-workload-certs\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x\" (UID: \"d4370a67-835f-4d52-97c4-0419593f7975\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:05.273355 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.273290    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4370a67-835f-4d52-97c4-0419593f7975-istio-token\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x\" (UID: \"d4370a67-835f-4d52-97c4-0419593f7975\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:05.374514 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.374480    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2w6b\" (UniqueName: \"kubernetes.io/projected/d4370a67-835f-4d52-97c4-0419593f7975-kube-api-access-l2w6b\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x\" (UID: \"d4370a67-835f-4d52-97c4-0419593f7975\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:05.374723 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.374521    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4370a67-835f-4d52-97c4-0419593f7975-workload-certs\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x\" (UID: \"d4370a67-835f-4d52-97c4-0419593f7975\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:05.374723 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.374586    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4370a67-835f-4d52-97c4-0419593f7975-istio-token\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x\" (UID: \"d4370a67-835f-4d52-97c4-0419593f7975\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:05.374723 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.374648    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4370a67-835f-4d52-97c4-0419593f7975-istio-podinfo\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x\" (UID: \"d4370a67-835f-4d52-97c4-0419593f7975\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:05.374723 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.374674    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4370a67-835f-4d52-97c4-0419593f7975-istio-data\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x\" (UID: \"d4370a67-835f-4d52-97c4-0419593f7975\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:05.374723 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.374717    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4370a67-835f-4d52-97c4-0419593f7975-istio-envoy\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x\" (UID: \"d4370a67-835f-4d52-97c4-0419593f7975\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:05.374953 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.374740    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4370a67-835f-4d52-97c4-0419593f7975-credential-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x\" (UID: \"d4370a67-835f-4d52-97c4-0419593f7975\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:05.374953 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.374766    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4370a67-835f-4d52-97c4-0419593f7975-istiod-ca-cert\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x\" (UID: \"d4370a67-835f-4d52-97c4-0419593f7975\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:05.374953 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.374806    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4370a67-835f-4d52-97c4-0419593f7975-workload-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x\" (UID: \"d4370a67-835f-4d52-97c4-0419593f7975\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:05.375249 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.375220    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4370a67-835f-4d52-97c4-0419593f7975-workload-certs\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x\" (UID: \"d4370a67-835f-4d52-97c4-0419593f7975\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:05.375339 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.375305    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4370a67-835f-4d52-97c4-0419593f7975-workload-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x\" (UID: \"d4370a67-835f-4d52-97c4-0419593f7975\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:05.375684 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.375650    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4370a67-835f-4d52-97c4-0419593f7975-istio-data\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x\" (UID: \"d4370a67-835f-4d52-97c4-0419593f7975\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:05.375932 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.375905    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4370a67-835f-4d52-97c4-0419593f7975-credential-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x\" (UID: \"d4370a67-835f-4d52-97c4-0419593f7975\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:05.376037 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.375948    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4370a67-835f-4d52-97c4-0419593f7975-istiod-ca-cert\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x\" (UID: \"d4370a67-835f-4d52-97c4-0419593f7975\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:05.377583 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.377559    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4370a67-835f-4d52-97c4-0419593f7975-istio-envoy\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x\" (UID: \"d4370a67-835f-4d52-97c4-0419593f7975\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:05.378093 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.378056    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4370a67-835f-4d52-97c4-0419593f7975-istio-podinfo\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x\" (UID: \"d4370a67-835f-4d52-97c4-0419593f7975\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:05.397603 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.397533    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4370a67-835f-4d52-97c4-0419593f7975-istio-token\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x\" (UID: \"d4370a67-835f-4d52-97c4-0419593f7975\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:05.397831 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.397809    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2w6b\" (UniqueName: \"kubernetes.io/projected/d4370a67-835f-4d52-97c4-0419593f7975-kube-api-access-l2w6b\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x\" (UID: \"d4370a67-835f-4d52-97c4-0419593f7975\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:05.464243 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.464213    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:05.790856 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.790831    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"]
Oct 30 00:43:05.793001 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:43:05.792975    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4370a67_835f_4d52_97c4_0419593f7975.slice/crio-aab546392622c12ca1dbda890ab241f7fee598daf9efedcec8d638ffb5b62ba4 WatchSource:0}: Error finding container aab546392622c12ca1dbda890ab241f7fee598daf9efedcec8d638ffb5b62ba4: Status 404 returned error can't find the container with id aab546392622c12ca1dbda890ab241f7fee598daf9efedcec8d638ffb5b62ba4
Oct 30 00:43:05.795347 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.795313    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:43:05.795450 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.795403    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:43:05.795523 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:05.795472    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:43:06.115454 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:06.115354    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x" event={"ID":"d4370a67-835f-4d52-97c4-0419593f7975","Type":"ContainerStarted","Data":"7c0f72c0435c5339acec126106d9a6d5cd3b794822b7605714567177c74a3621"}
Oct 30 00:43:06.115454 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:06.115414    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x" event={"ID":"d4370a67-835f-4d52-97c4-0419593f7975","Type":"ContainerStarted","Data":"aab546392622c12ca1dbda890ab241f7fee598daf9efedcec8d638ffb5b62ba4"}
Oct 30 00:43:06.160460 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:06.160409    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x" podStartSLOduration=1.16039303 podStartE2EDuration="1.16039303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:06.158301346 +0000 UTC m=+676.545977734" watchObservedRunningTime="2026-10-30 00:43:06.16039303 +0000 UTC m=+676.548069418"
Oct 30 00:43:06.464971 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:06.464931    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:06.470756 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:06.470732    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:07.120610 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:07.120580    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:07.121620 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:07.121598    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:09.965697 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:09.965670    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp"]
Oct 30 00:43:09.968372 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:09.968347    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp"
Oct 30 00:43:09.998132 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:09.998103    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-cefb27-serving-cert\""
Oct 30 00:43:10.001619 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.001594    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp"]
Oct 30 00:43:10.024247 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.024211    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j6zd\" (UniqueName: \"kubernetes.io/projected/77896a6a-93c0-4f84-91ff-676e591f4453-kube-api-access-2j6zd\") pod \"maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp\" (UID: \"77896a6a-93c0-4f84-91ff-676e591f4453\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp"
Oct 30 00:43:10.024435 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.024280    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77896a6a-93c0-4f84-91ff-676e591f4453-metrics-certs\") pod \"maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp\" (UID: \"77896a6a-93c0-4f84-91ff-676e591f4453\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp"
Oct 30 00:43:10.024522 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.024438    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77896a6a-93c0-4f84-91ff-676e591f4453-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp\" (UID: \"77896a6a-93c0-4f84-91ff-676e591f4453\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp"
Oct 30 00:43:10.024522 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.024482    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77896a6a-93c0-4f84-91ff-676e591f4453-maas-api-tls\") pod \"maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp\" (UID: \"77896a6a-93c0-4f84-91ff-676e591f4453\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp"
Oct 30 00:43:10.125304 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.125273    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77896a6a-93c0-4f84-91ff-676e591f4453-maas-api-tls\") pod \"maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp\" (UID: \"77896a6a-93c0-4f84-91ff-676e591f4453\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp"
Oct 30 00:43:10.125498 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.125387    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j6zd\" (UniqueName: \"kubernetes.io/projected/77896a6a-93c0-4f84-91ff-676e591f4453-kube-api-access-2j6zd\") pod \"maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp\" (UID: \"77896a6a-93c0-4f84-91ff-676e591f4453\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp"
Oct 30 00:43:10.125498 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.125420    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77896a6a-93c0-4f84-91ff-676e591f4453-metrics-certs\") pod \"maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp\" (UID: \"77896a6a-93c0-4f84-91ff-676e591f4453\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp"
Oct 30 00:43:10.125498 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.125454    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77896a6a-93c0-4f84-91ff-676e591f4453-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp\" (UID: \"77896a6a-93c0-4f84-91ff-676e591f4453\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp"
Oct 30 00:43:10.126139 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.126084    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77896a6a-93c0-4f84-91ff-676e591f4453-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp\" (UID: \"77896a6a-93c0-4f84-91ff-676e591f4453\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp"
Oct 30 00:43:10.128038 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.128006    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77896a6a-93c0-4f84-91ff-676e591f4453-maas-api-tls\") pod \"maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp\" (UID: \"77896a6a-93c0-4f84-91ff-676e591f4453\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp"
Oct 30 00:43:10.128038 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.128032    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77896a6a-93c0-4f84-91ff-676e591f4453-metrics-certs\") pod \"maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp\" (UID: \"77896a6a-93c0-4f84-91ff-676e591f4453\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp"
Oct 30 00:43:10.150969 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.150944    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j6zd\" (UniqueName: \"kubernetes.io/projected/77896a6a-93c0-4f84-91ff-676e591f4453-kube-api-access-2j6zd\") pod \"maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp\" (UID: \"77896a6a-93c0-4f84-91ff-676e591f4453\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp"
Oct 30 00:43:10.239217 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.239129    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-cefb27-dfdc89d75-jm8hc"]
Oct 30 00:43:10.241678 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.241662    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cefb27-dfdc89d75-jm8hc"
Oct 30 00:43:10.260937 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.260914    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-cefb27\""
Oct 30 00:43:10.260937 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.261023    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-cefb27-dockercfg-pfc9d\""
Oct 30 00:43:10.264143 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.264118    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-cefb27-dfdc89d75-jm8hc"]
Oct 30 00:43:10.282255 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.282231    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp"
Oct 30 00:43:10.328896 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.327227    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-cefb27-88db76d58-8gzgh"]
Oct 30 00:43:10.356279 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.336351    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/691e02fc-7fd6-454b-a78e-31e3c34175c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-cefb27-dfdc89d75-jm8hc\" (UID: \"691e02fc-7fd6-454b-a78e-31e3c34175c3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cefb27-dfdc89d75-jm8hc"
Oct 30 00:43:10.356279 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.336557    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2hlb\" (UniqueName: \"kubernetes.io/projected/691e02fc-7fd6-454b-a78e-31e3c34175c3-kube-api-access-h2hlb\") pod \"payload-pre-processing-e2e-worker-w3-cefb27-dfdc89d75-jm8hc\" (UID: \"691e02fc-7fd6-454b-a78e-31e3c34175c3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cefb27-dfdc89d75-jm8hc"
Oct 30 00:43:10.367604 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.367563    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-cefb27-88db76d58-8gzgh"
Oct 30 00:43:10.371307 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.371273    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-cefb27-88db76d58-8gzgh"]
Oct 30 00:43:10.438635 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.438597    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8x55\" (UniqueName: \"kubernetes.io/projected/20350585-8ccb-4f19-a634-c2cc4031b330-kube-api-access-z8x55\") pod \"payload-processing-e2e-worker-w3-cefb27-88db76d58-8gzgh\" (UID: \"20350585-8ccb-4f19-a634-c2cc4031b330\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-cefb27-88db76d58-8gzgh"
Oct 30 00:43:10.438825 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.438764    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20350585-8ccb-4f19-a634-c2cc4031b330-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-cefb27-88db76d58-8gzgh\" (UID: \"20350585-8ccb-4f19-a634-c2cc4031b330\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-cefb27-88db76d58-8gzgh"
Oct 30 00:43:10.438878 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.438847    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/691e02fc-7fd6-454b-a78e-31e3c34175c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-cefb27-dfdc89d75-jm8hc\" (UID: \"691e02fc-7fd6-454b-a78e-31e3c34175c3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cefb27-dfdc89d75-jm8hc"
Oct 30 00:43:10.438913 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.438878    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2hlb\" (UniqueName: \"kubernetes.io/projected/691e02fc-7fd6-454b-a78e-31e3c34175c3-kube-api-access-h2hlb\") pod \"payload-pre-processing-e2e-worker-w3-cefb27-dfdc89d75-jm8hc\" (UID: \"691e02fc-7fd6-454b-a78e-31e3c34175c3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cefb27-dfdc89d75-jm8hc"
Oct 30 00:43:10.439511 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.439493    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/691e02fc-7fd6-454b-a78e-31e3c34175c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-cefb27-dfdc89d75-jm8hc\" (UID: \"691e02fc-7fd6-454b-a78e-31e3c34175c3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cefb27-dfdc89d75-jm8hc"
Oct 30 00:43:10.467827 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.467771    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2hlb\" (UniqueName: \"kubernetes.io/projected/691e02fc-7fd6-454b-a78e-31e3c34175c3-kube-api-access-h2hlb\") pod \"payload-pre-processing-e2e-worker-w3-cefb27-dfdc89d75-jm8hc\" (UID: \"691e02fc-7fd6-454b-a78e-31e3c34175c3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cefb27-dfdc89d75-jm8hc"
Oct 30 00:43:10.539502 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.539433    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20350585-8ccb-4f19-a634-c2cc4031b330-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-cefb27-88db76d58-8gzgh\" (UID: \"20350585-8ccb-4f19-a634-c2cc4031b330\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-cefb27-88db76d58-8gzgh"
Oct 30 00:43:10.539502 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.539494    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8x55\" (UniqueName: \"kubernetes.io/projected/20350585-8ccb-4f19-a634-c2cc4031b330-kube-api-access-z8x55\") pod \"payload-processing-e2e-worker-w3-cefb27-88db76d58-8gzgh\" (UID: \"20350585-8ccb-4f19-a634-c2cc4031b330\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-cefb27-88db76d58-8gzgh"
Oct 30 00:43:10.540377 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.540352    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20350585-8ccb-4f19-a634-c2cc4031b330-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-cefb27-88db76d58-8gzgh\" (UID: \"20350585-8ccb-4f19-a634-c2cc4031b330\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-cefb27-88db76d58-8gzgh"
Oct 30 00:43:10.554325 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.554302    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cefb27-dfdc89d75-jm8hc"
Oct 30 00:43:10.562385 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.562363    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8x55\" (UniqueName: \"kubernetes.io/projected/20350585-8ccb-4f19-a634-c2cc4031b330-kube-api-access-z8x55\") pod \"payload-processing-e2e-worker-w3-cefb27-88db76d58-8gzgh\" (UID: \"20350585-8ccb-4f19-a634-c2cc4031b330\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-cefb27-88db76d58-8gzgh"
Oct 30 00:43:10.636475 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.636445    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp"]
Oct 30 00:43:10.638260 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:43:10.638230    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77896a6a_93c0_4f84_91ff_676e591f4453.slice/crio-4f19f0f1651ea06b4519f80c86017ca619b39839059f7ed9fc66847e9a06e480 WatchSource:0}: Error finding container 4f19f0f1651ea06b4519f80c86017ca619b39839059f7ed9fc66847e9a06e480: Status 404 returned error can't find the container with id 4f19f0f1651ea06b4519f80c86017ca619b39839059f7ed9fc66847e9a06e480
Oct 30 00:43:10.639739 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.639717    4393 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:43:10.689677 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.689643    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-cefb27-88db76d58-8gzgh"
Oct 30 00:43:10.868690 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.868668    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-cefb27-dfdc89d75-jm8hc"]
Oct 30 00:43:10.871405 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:43:10.871372    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod691e02fc_7fd6_454b_a78e_31e3c34175c3.slice/crio-bcb7c68e5438240d2dabe844e2502ccfcb768349e409489bda89f881492a3301 WatchSource:0}: Error finding container bcb7c68e5438240d2dabe844e2502ccfcb768349e409489bda89f881492a3301: Status 404 returned error can't find the container with id bcb7c68e5438240d2dabe844e2502ccfcb768349e409489bda89f881492a3301
Oct 30 00:43:10.988138 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:10.988116    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-cefb27-88db76d58-8gzgh"]
Oct 30 00:43:10.989505 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:43:10.989483    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20350585_8ccb_4f19_a634_c2cc4031b330.slice/crio-f84fbc7066ad275a6c8a5cf6b3977c13d81f6d1a144c4cff3ab6387e05143a70 WatchSource:0}: Error finding container f84fbc7066ad275a6c8a5cf6b3977c13d81f6d1a144c4cff3ab6387e05143a70: Status 404 returned error can't find the container with id f84fbc7066ad275a6c8a5cf6b3977c13d81f6d1a144c4cff3ab6387e05143a70
Oct 30 00:43:11.147453 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:11.147418    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp" event={"ID":"77896a6a-93c0-4f84-91ff-676e591f4453","Type":"ContainerStarted","Data":"f5762bc14c89742b92ea2181a18990e77211baadc310e1a43f753f95e80020f0"}
Oct 30 00:43:11.147612 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:11.147473    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp" event={"ID":"77896a6a-93c0-4f84-91ff-676e591f4453","Type":"ContainerStarted","Data":"4f19f0f1651ea06b4519f80c86017ca619b39839059f7ed9fc66847e9a06e480"}
Oct 30 00:43:11.147612 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:11.147573    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp"
Oct 30 00:43:11.149338 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:11.149305    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-cefb27-88db76d58-8gzgh" event={"ID":"20350585-8ccb-4f19-a634-c2cc4031b330","Type":"ContainerStarted","Data":"d3f57a045e5068597119a4ee91b74a93276b8686798004d37dc39d649d67d205"}
Oct 30 00:43:11.149338 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:11.149352    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-cefb27-88db76d58-8gzgh" event={"ID":"20350585-8ccb-4f19-a634-c2cc4031b330","Type":"ContainerStarted","Data":"f84fbc7066ad275a6c8a5cf6b3977c13d81f6d1a144c4cff3ab6387e05143a70"}
Oct 30 00:43:11.149735 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:11.149718    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-cefb27-88db76d58-8gzgh"
Oct 30 00:43:11.151334 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:11.151308    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cefb27-dfdc89d75-jm8hc" event={"ID":"691e02fc-7fd6-454b-a78e-31e3c34175c3","Type":"ContainerStarted","Data":"b299afbb36e5c2baf69aed5d23019a145aa46c609984714ea3893de90d4b5a80"}
Oct 30 00:43:11.151445 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:11.151357    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cefb27-dfdc89d75-jm8hc" event={"ID":"691e02fc-7fd6-454b-a78e-31e3c34175c3","Type":"ContainerStarted","Data":"bcb7c68e5438240d2dabe844e2502ccfcb768349e409489bda89f881492a3301"}
Oct 30 00:43:11.151445 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:11.151392    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cefb27-dfdc89d75-jm8hc"
Oct 30 00:43:11.189439 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:11.189390    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp" podStartSLOduration=1.8103441820000001 podStartE2EDuration="2.189376857s" totalImagesPullingTime="379.032675ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:11.187421611 +0000 UTC m=+681.575098002" watchObservedRunningTime="2026-10-30 00:43:11.189376857 +0000 UTC m=+681.577053246"
Oct 30 00:43:11.227253 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:11.227200    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cefb27-dfdc89d75-jm8hc" podStartSLOduration=1.227185207 podStartE2EDuration="1.227185207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:11.226491918 +0000 UTC m=+681.614168307" watchObservedRunningTime="2026-10-30 00:43:11.227185207 +0000 UTC m=+681.614861744"
Oct 30 00:43:11.265861 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:11.265818    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-cefb27-88db76d58-8gzgh" podStartSLOduration=1.265805252 podStartE2EDuration="1.265805252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:11.265630007 +0000 UTC m=+681.653306397" watchObservedRunningTime="2026-10-30 00:43:11.265805252 +0000 UTC m=+681.653481672"
Oct 30 00:43:11.725710 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:11.725675    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462"]
Oct 30 00:43:11.729501 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:11.729480    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462"
Oct 30 00:43:11.747123 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:11.747093    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-6f4fba-serving-cert\""
Oct 30 00:43:11.756973 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:11.756942    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462"]
Oct 30 00:43:11.856290 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:11.856256    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38092938-4c3d-41ae-b7dd-25ddd1869d7d-metrics-certs\") pod \"maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462\" (UID: \"38092938-4c3d-41ae-b7dd-25ddd1869d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462"
Oct 30 00:43:11.856452 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:11.856361    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38092938-4c3d-41ae-b7dd-25ddd1869d7d-maas-api-tls\") pod \"maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462\" (UID: \"38092938-4c3d-41ae-b7dd-25ddd1869d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462"
Oct 30 00:43:11.856515 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:11.856463    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sqnc\" (UniqueName: \"kubernetes.io/projected/38092938-4c3d-41ae-b7dd-25ddd1869d7d-kube-api-access-2sqnc\") pod \"maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462\" (UID: \"38092938-4c3d-41ae-b7dd-25ddd1869d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462"
Oct 30 00:43:11.856574 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:11.856511    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38092938-4c3d-41ae-b7dd-25ddd1869d7d-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462\" (UID: \"38092938-4c3d-41ae-b7dd-25ddd1869d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462"
Oct 30 00:43:11.957924 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:11.957890    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38092938-4c3d-41ae-b7dd-25ddd1869d7d-metrics-certs\") pod \"maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462\" (UID: \"38092938-4c3d-41ae-b7dd-25ddd1869d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462"
Oct 30 00:43:11.958112 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:11.957935    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38092938-4c3d-41ae-b7dd-25ddd1869d7d-maas-api-tls\") pod \"maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462\" (UID: \"38092938-4c3d-41ae-b7dd-25ddd1869d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462"
Oct 30 00:43:11.958112 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:11.958002    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sqnc\" (UniqueName: \"kubernetes.io/projected/38092938-4c3d-41ae-b7dd-25ddd1869d7d-kube-api-access-2sqnc\") pod \"maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462\" (UID: \"38092938-4c3d-41ae-b7dd-25ddd1869d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462"
Oct 30 00:43:11.958112 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:11.958029    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38092938-4c3d-41ae-b7dd-25ddd1869d7d-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462\" (UID: \"38092938-4c3d-41ae-b7dd-25ddd1869d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462"
Oct 30 00:43:11.958285 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:43:11.958119    4393 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-6f4fba-serving-cert: secret "maas-api-e2e-worker-w5-6f4fba-serving-cert" not found
Oct 30 00:43:11.958285 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:43:11.958202    4393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38092938-4c3d-41ae-b7dd-25ddd1869d7d-maas-api-tls podName:38092938-4c3d-41ae-b7dd-25ddd1869d7d nodeName:}" failed. No retries permitted until 2026-10-30 00:43:12.458178218 +0000 UTC m=+682.845854613 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/38092938-4c3d-41ae-b7dd-25ddd1869d7d-maas-api-tls") pod "maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462" (UID: "38092938-4c3d-41ae-b7dd-25ddd1869d7d") : secret "maas-api-e2e-worker-w5-6f4fba-serving-cert" not found
Oct 30 00:43:11.958944 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:11.958919    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38092938-4c3d-41ae-b7dd-25ddd1869d7d-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462\" (UID: \"38092938-4c3d-41ae-b7dd-25ddd1869d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462"
Oct 30 00:43:11.960504 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:11.960484    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38092938-4c3d-41ae-b7dd-25ddd1869d7d-metrics-certs\") pod \"maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462\" (UID: \"38092938-4c3d-41ae-b7dd-25ddd1869d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462"
Oct 30 00:43:11.983962 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:11.983905    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sqnc\" (UniqueName: \"kubernetes.io/projected/38092938-4c3d-41ae-b7dd-25ddd1869d7d-kube-api-access-2sqnc\") pod \"maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462\" (UID: \"38092938-4c3d-41ae-b7dd-25ddd1869d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462"
Oct 30 00:43:12.040027 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:12.039996    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-6f4fba-5c479c7dd7-9sxbh"]
Oct 30 00:43:12.043753 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:12.043730    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6f4fba-5c479c7dd7-9sxbh"
Oct 30 00:43:12.060904 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:12.060879    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-6f4fba-dockercfg-gjlp6\""
Oct 30 00:43:12.061048 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:12.060929    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-6f4fba\""
Oct 30 00:43:12.063822 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:12.063799    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-6f4fba-5c479c7dd7-9sxbh"]
Oct 30 00:43:12.142764 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:12.142538    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-6f4fba-86877b49f8-n59lf"]
Oct 30 00:43:12.147717 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:12.147687    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-6f4fba-86877b49f8-n59lf"
Oct 30 00:43:12.168867 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:12.162570    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44688868-ab51-4384-9887-b1d61810eca3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-6f4fba-5c479c7dd7-9sxbh\" (UID: \"44688868-ab51-4384-9887-b1d61810eca3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6f4fba-5c479c7dd7-9sxbh"
Oct 30 00:43:12.168867 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:12.162622    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnjnt\" (UniqueName: \"kubernetes.io/projected/44688868-ab51-4384-9887-b1d61810eca3-kube-api-access-fnjnt\") pod \"payload-pre-processing-e2e-worker-w5-6f4fba-5c479c7dd7-9sxbh\" (UID: \"44688868-ab51-4384-9887-b1d61810eca3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6f4fba-5c479c7dd7-9sxbh"
Oct 30 00:43:12.171623 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:12.171521    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-6f4fba-86877b49f8-n59lf"]
Oct 30 00:43:12.263465 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:12.263380    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44688868-ab51-4384-9887-b1d61810eca3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-6f4fba-5c479c7dd7-9sxbh\" (UID: \"44688868-ab51-4384-9887-b1d61810eca3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6f4fba-5c479c7dd7-9sxbh"
Oct 30 00:43:12.263465 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:12.263417    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnjnt\" (UniqueName: \"kubernetes.io/projected/44688868-ab51-4384-9887-b1d61810eca3-kube-api-access-fnjnt\") pod \"payload-pre-processing-e2e-worker-w5-6f4fba-5c479c7dd7-9sxbh\" (UID: \"44688868-ab51-4384-9887-b1d61810eca3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6f4fba-5c479c7dd7-9sxbh"
Oct 30 00:43:12.263465 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:12.263460    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7czbv\" (UniqueName: \"kubernetes.io/projected/2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd-kube-api-access-7czbv\") pod \"payload-processing-e2e-worker-w5-6f4fba-86877b49f8-n59lf\" (UID: \"2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-6f4fba-86877b49f8-n59lf"
Oct 30 00:43:12.264035 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:12.264006    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-6f4fba-86877b49f8-n59lf\" (UID: \"2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-6f4fba-86877b49f8-n59lf"
Oct 30 00:43:12.264278 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:12.264256    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44688868-ab51-4384-9887-b1d61810eca3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-6f4fba-5c479c7dd7-9sxbh\" (UID: \"44688868-ab51-4384-9887-b1d61810eca3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6f4fba-5c479c7dd7-9sxbh"
Oct 30 00:43:12.287238 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:12.287205    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnjnt\" (UniqueName: \"kubernetes.io/projected/44688868-ab51-4384-9887-b1d61810eca3-kube-api-access-fnjnt\") pod \"payload-pre-processing-e2e-worker-w5-6f4fba-5c479c7dd7-9sxbh\" (UID: \"44688868-ab51-4384-9887-b1d61810eca3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6f4fba-5c479c7dd7-9sxbh"
Oct 30 00:43:12.356954 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:12.356925    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6f4fba-5c479c7dd7-9sxbh"
Oct 30 00:43:12.365290 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:12.365250    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-6f4fba-86877b49f8-n59lf\" (UID: \"2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-6f4fba-86877b49f8-n59lf"
Oct 30 00:43:12.365434 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:12.365346    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7czbv\" (UniqueName: \"kubernetes.io/projected/2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd-kube-api-access-7czbv\") pod \"payload-processing-e2e-worker-w5-6f4fba-86877b49f8-n59lf\" (UID: \"2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-6f4fba-86877b49f8-n59lf"
Oct 30 00:43:12.365949 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:12.365931    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-6f4fba-86877b49f8-n59lf\" (UID: \"2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-6f4fba-86877b49f8-n59lf"
Oct 30 00:43:12.388227 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:12.388200    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7czbv\" (UniqueName: \"kubernetes.io/projected/2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd-kube-api-access-7czbv\") pod \"payload-processing-e2e-worker-w5-6f4fba-86877b49f8-n59lf\" (UID: \"2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-6f4fba-86877b49f8-n59lf"
Oct 30 00:43:12.466558 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:12.466523    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38092938-4c3d-41ae-b7dd-25ddd1869d7d-maas-api-tls\") pod \"maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462\" (UID: \"38092938-4c3d-41ae-b7dd-25ddd1869d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462"
Oct 30 00:43:12.469066 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:12.469044    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38092938-4c3d-41ae-b7dd-25ddd1869d7d-maas-api-tls\") pod \"maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462\" (UID: \"38092938-4c3d-41ae-b7dd-25ddd1869d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462"
Oct 30 00:43:12.481944 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:12.481923    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-6f4fba-86877b49f8-n59lf"
Oct 30 00:43:12.644888 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:12.644861    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462"
Oct 30 00:43:12.665325 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:12.663552    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-6f4fba-5c479c7dd7-9sxbh"]
Oct 30 00:43:12.783116 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:12.783052    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-6f4fba-86877b49f8-n59lf"]
Oct 30 00:43:12.784598 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:43:12.784575    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fd09a90_3f1e_4fc9_9ff8_fd8bab9a22cd.slice/crio-9b392dfd47b6ee4a8c7b53ad524baca294d22e44c1acfabbcf94bf6cff183833 WatchSource:0}: Error finding container 9b392dfd47b6ee4a8c7b53ad524baca294d22e44c1acfabbcf94bf6cff183833: Status 404 returned error can't find the container with id 9b392dfd47b6ee4a8c7b53ad524baca294d22e44c1acfabbcf94bf6cff183833
Oct 30 00:43:12.964154 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:12.964123    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462"]
Oct 30 00:43:12.965051 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:43:12.965028    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38092938_4c3d_41ae_b7dd_25ddd1869d7d.slice/crio-b39d38d7428de526834f1d3d69c2a466f8619f5f26ebd20c8e7dc785171c170a WatchSource:0}: Error finding container b39d38d7428de526834f1d3d69c2a466f8619f5f26ebd20c8e7dc785171c170a: Status 404 returned error can't find the container with id b39d38d7428de526834f1d3d69c2a466f8619f5f26ebd20c8e7dc785171c170a
Oct 30 00:43:12.985789 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:12.985758    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ca5363-57d48d9d99-ctxwr"]
Oct 30 00:43:12.989724 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:12.989704    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca5363-57d48d9d99-ctxwr"
Oct 30 00:43:13.009056 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.009029    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-ca5363\""
Oct 30 00:43:13.009185 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.009083    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-ca5363-dockercfg-zvclm\""
Oct 30 00:43:13.010556 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.010533    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ca5363-57d48d9d99-ctxwr"]
Oct 30 00:43:13.066585 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.066494    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ca5363-657fbd45cb-wc7w2"]
Oct 30 00:43:13.073495 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.073460    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7zwp\" (UniqueName: \"kubernetes.io/projected/0fa2bddd-6b15-47a8-95cc-8f952594e284-kube-api-access-z7zwp\") pod \"payload-pre-processing-e2e-worker-w4-ca5363-57d48d9d99-ctxwr\" (UID: \"0fa2bddd-6b15-47a8-95cc-8f952594e284\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca5363-57d48d9d99-ctxwr"
Oct 30 00:43:13.073757 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.073574    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa2bddd-6b15-47a8-95cc-8f952594e284-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ca5363-57d48d9d99-ctxwr\" (UID: \"0fa2bddd-6b15-47a8-95cc-8f952594e284\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca5363-57d48d9d99-ctxwr"
Oct 30 00:43:13.076429 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.076402    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ca5363-657fbd45cb-wc7w2"
Oct 30 00:43:13.092085 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.092059    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ca5363-657fbd45cb-wc7w2"]
Oct 30 00:43:13.174943 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.174907    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rngzf\" (UniqueName: \"kubernetes.io/projected/dd52771e-289e-4bd7-952e-eaddd165eb58-kube-api-access-rngzf\") pod \"payload-processing-e2e-worker-w4-ca5363-657fbd45cb-wc7w2\" (UID: \"dd52771e-289e-4bd7-952e-eaddd165eb58\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ca5363-657fbd45cb-wc7w2"
Oct 30 00:43:13.175116 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.174948    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7zwp\" (UniqueName: \"kubernetes.io/projected/0fa2bddd-6b15-47a8-95cc-8f952594e284-kube-api-access-z7zwp\") pod \"payload-pre-processing-e2e-worker-w4-ca5363-57d48d9d99-ctxwr\" (UID: \"0fa2bddd-6b15-47a8-95cc-8f952594e284\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca5363-57d48d9d99-ctxwr"
Oct 30 00:43:13.175116 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.175073    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa2bddd-6b15-47a8-95cc-8f952594e284-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ca5363-57d48d9d99-ctxwr\" (UID: \"0fa2bddd-6b15-47a8-95cc-8f952594e284\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca5363-57d48d9d99-ctxwr"
Oct 30 00:43:13.175116 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.175108    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd52771e-289e-4bd7-952e-eaddd165eb58-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-ca5363-657fbd45cb-wc7w2\" (UID: \"dd52771e-289e-4bd7-952e-eaddd165eb58\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ca5363-657fbd45cb-wc7w2"
Oct 30 00:43:13.176162 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.176137    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa2bddd-6b15-47a8-95cc-8f952594e284-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ca5363-57d48d9d99-ctxwr\" (UID: \"0fa2bddd-6b15-47a8-95cc-8f952594e284\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca5363-57d48d9d99-ctxwr"
Oct 30 00:43:13.182116 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.182082    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-6f4fba-86877b49f8-n59lf" event={"ID":"2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd","Type":"ContainerStarted","Data":"534690997e7276a0920e37caa6a65c204cc478d8b80679c92581c09f1602647a"}
Oct 30 00:43:13.182233 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.182148    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-6f4fba-86877b49f8-n59lf"
Oct 30 00:43:13.182233 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.182171    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-6f4fba-86877b49f8-n59lf" event={"ID":"2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd","Type":"ContainerStarted","Data":"9b392dfd47b6ee4a8c7b53ad524baca294d22e44c1acfabbcf94bf6cff183833"}
Oct 30 00:43:13.183333 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.183311    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462" event={"ID":"38092938-4c3d-41ae-b7dd-25ddd1869d7d","Type":"ContainerStarted","Data":"b39d38d7428de526834f1d3d69c2a466f8619f5f26ebd20c8e7dc785171c170a"}
Oct 30 00:43:13.185007 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.184983    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6f4fba-5c479c7dd7-9sxbh" event={"ID":"44688868-ab51-4384-9887-b1d61810eca3","Type":"ContainerStarted","Data":"87814af2e15bf8ebe57f4f8606f4f7fa5a87f4e1b13698355bae99dd53f47e09"}
Oct 30 00:43:13.185007 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.185030    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6f4fba-5c479c7dd7-9sxbh" event={"ID":"44688868-ab51-4384-9887-b1d61810eca3","Type":"ContainerStarted","Data":"8cc9e2bc7ea42b3cbbcbb8929226ca0e3baee4b3762917175931e19c0c68a7bb"}
Oct 30 00:43:13.185221 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.185148    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6f4fba-5c479c7dd7-9sxbh"
Oct 30 00:43:13.197761 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.197741    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7zwp\" (UniqueName: \"kubernetes.io/projected/0fa2bddd-6b15-47a8-95cc-8f952594e284-kube-api-access-z7zwp\") pod \"payload-pre-processing-e2e-worker-w4-ca5363-57d48d9d99-ctxwr\" (UID: \"0fa2bddd-6b15-47a8-95cc-8f952594e284\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca5363-57d48d9d99-ctxwr"
Oct 30 00:43:13.223372 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.223319    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-6f4fba-86877b49f8-n59lf" podStartSLOduration=1.223303284 podStartE2EDuration="1.223303284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:13.220121816 +0000 UTC m=+683.607798207" watchObservedRunningTime="2026-10-30 00:43:13.223303284 +0000 UTC m=+683.610979673"
Oct 30 00:43:13.260852 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.260814    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6f4fba-5c479c7dd7-9sxbh" podStartSLOduration=1.260800767 podStartE2EDuration="1.260800767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:13.260638506 +0000 UTC m=+683.648314896" watchObservedRunningTime="2026-10-30 00:43:13.260800767 +0000 UTC m=+683.648477155"
Oct 30 00:43:13.276200 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.276175    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd52771e-289e-4bd7-952e-eaddd165eb58-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-ca5363-657fbd45cb-wc7w2\" (UID: \"dd52771e-289e-4bd7-952e-eaddd165eb58\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ca5363-657fbd45cb-wc7w2"
Oct 30 00:43:13.276311 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.276286    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rngzf\" (UniqueName: \"kubernetes.io/projected/dd52771e-289e-4bd7-952e-eaddd165eb58-kube-api-access-rngzf\") pod \"payload-processing-e2e-worker-w4-ca5363-657fbd45cb-wc7w2\" (UID: \"dd52771e-289e-4bd7-952e-eaddd165eb58\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ca5363-657fbd45cb-wc7w2"
Oct 30 00:43:13.276962 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.276940    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd52771e-289e-4bd7-952e-eaddd165eb58-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-ca5363-657fbd45cb-wc7w2\" (UID: \"dd52771e-289e-4bd7-952e-eaddd165eb58\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ca5363-657fbd45cb-wc7w2"
Oct 30 00:43:13.298451 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.298426    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rngzf\" (UniqueName: \"kubernetes.io/projected/dd52771e-289e-4bd7-952e-eaddd165eb58-kube-api-access-rngzf\") pod \"payload-processing-e2e-worker-w4-ca5363-657fbd45cb-wc7w2\" (UID: \"dd52771e-289e-4bd7-952e-eaddd165eb58\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ca5363-657fbd45cb-wc7w2"
Oct 30 00:43:13.303793 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.303599    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca5363-57d48d9d99-ctxwr"
Oct 30 00:43:13.408345 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.408088    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ca5363-657fbd45cb-wc7w2"
Oct 30 00:43:13.608686 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.608656    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ca5363-57d48d9d99-ctxwr"]
Oct 30 00:43:13.609857 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:43:13.609831    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa2bddd_6b15_47a8_95cc_8f952594e284.slice/crio-0b2ed3b08df987477a5538f8c4195463b26622429716dba9ab5b3226ffec3752 WatchSource:0}: Error finding container 0b2ed3b08df987477a5538f8c4195463b26622429716dba9ab5b3226ffec3752: Status 404 returned error can't find the container with id 0b2ed3b08df987477a5538f8c4195463b26622429716dba9ab5b3226ffec3752
Oct 30 00:43:13.737184 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.737163    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ca5363-657fbd45cb-wc7w2"]
Oct 30 00:43:13.739411 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:43:13.739382    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd52771e_289e_4bd7_952e_eaddd165eb58.slice/crio-520e946f022b988a79a61dfcb0edceb643e7c6143ff535a8c25dfb296e450cb5 WatchSource:0}: Error finding container 520e946f022b988a79a61dfcb0edceb643e7c6143ff535a8c25dfb296e450cb5: Status 404 returned error can't find the container with id 520e946f022b988a79a61dfcb0edceb643e7c6143ff535a8c25dfb296e450cb5
Oct 30 00:43:13.877634 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.877598    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-29be73-58c6d59956-2sszw"]
Oct 30 00:43:13.881506 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.881487    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-29be73-58c6d59956-2sszw"
Oct 30 00:43:13.899040 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.899020    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-29be73\""
Oct 30 00:43:13.907137 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.907115    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-29be73-58c6d59956-2sszw"]
Oct 30 00:43:13.936028 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.936001    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-29be73-64579fcddc-7cr2l"]
Oct 30 00:43:13.939996 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.939976    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29be73-64579fcddc-7cr2l"
Oct 30 00:43:13.959023 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.958995    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-29be73-dockercfg-xm672\""
Oct 30 00:43:13.962704 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.962682    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-29be73-64579fcddc-7cr2l"]
Oct 30 00:43:13.985288 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.985262    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zczhh\" (UniqueName: \"kubernetes.io/projected/7bd51483-6bfc-4535-bd66-4c4e74ca70c8-kube-api-access-zczhh\") pod \"payload-processing-e2e-worker-w1-29be73-58c6d59956-2sszw\" (UID: \"7bd51483-6bfc-4535-bd66-4c4e74ca70c8\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-29be73-58c6d59956-2sszw"
Oct 30 00:43:13.985411 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:13.985330    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd51483-6bfc-4535-bd66-4c4e74ca70c8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-29be73-58c6d59956-2sszw\" (UID: \"7bd51483-6bfc-4535-bd66-4c4e74ca70c8\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-29be73-58c6d59956-2sszw"
Oct 30 00:43:14.086519 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:14.086488    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zczhh\" (UniqueName: \"kubernetes.io/projected/7bd51483-6bfc-4535-bd66-4c4e74ca70c8-kube-api-access-zczhh\") pod \"payload-processing-e2e-worker-w1-29be73-58c6d59956-2sszw\" (UID: \"7bd51483-6bfc-4535-bd66-4c4e74ca70c8\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-29be73-58c6d59956-2sszw"
Oct 30 00:43:14.086972 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:14.086538    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/993c1f81-d8b2-46ac-8a86-f6399168de03-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-29be73-64579fcddc-7cr2l\" (UID: \"993c1f81-d8b2-46ac-8a86-f6399168de03\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29be73-64579fcddc-7cr2l"
Oct 30 00:43:14.086972 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:14.086583    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd51483-6bfc-4535-bd66-4c4e74ca70c8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-29be73-58c6d59956-2sszw\" (UID: \"7bd51483-6bfc-4535-bd66-4c4e74ca70c8\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-29be73-58c6d59956-2sszw"
Oct 30 00:43:14.086972 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:14.086648    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc4tb\" (UniqueName: \"kubernetes.io/projected/993c1f81-d8b2-46ac-8a86-f6399168de03-kube-api-access-pc4tb\") pod \"payload-pre-processing-e2e-worker-w1-29be73-64579fcddc-7cr2l\" (UID: \"993c1f81-d8b2-46ac-8a86-f6399168de03\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29be73-64579fcddc-7cr2l"
Oct 30 00:43:14.087299 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:14.087281    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd51483-6bfc-4535-bd66-4c4e74ca70c8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-29be73-58c6d59956-2sszw\" (UID: \"7bd51483-6bfc-4535-bd66-4c4e74ca70c8\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-29be73-58c6d59956-2sszw"
Oct 30 00:43:14.109721 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:14.109687    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zczhh\" (UniqueName: \"kubernetes.io/projected/7bd51483-6bfc-4535-bd66-4c4e74ca70c8-kube-api-access-zczhh\") pod \"payload-processing-e2e-worker-w1-29be73-58c6d59956-2sszw\" (UID: \"7bd51483-6bfc-4535-bd66-4c4e74ca70c8\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-29be73-58c6d59956-2sszw"
Oct 30 00:43:14.188188 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:14.188110    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc4tb\" (UniqueName: \"kubernetes.io/projected/993c1f81-d8b2-46ac-8a86-f6399168de03-kube-api-access-pc4tb\") pod \"payload-pre-processing-e2e-worker-w1-29be73-64579fcddc-7cr2l\" (UID: \"993c1f81-d8b2-46ac-8a86-f6399168de03\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29be73-64579fcddc-7cr2l"
Oct 30 00:43:14.188333 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:14.188226    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/993c1f81-d8b2-46ac-8a86-f6399168de03-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-29be73-64579fcddc-7cr2l\" (UID: \"993c1f81-d8b2-46ac-8a86-f6399168de03\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29be73-64579fcddc-7cr2l"
Oct 30 00:43:14.188936 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:14.188915    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/993c1f81-d8b2-46ac-8a86-f6399168de03-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-29be73-64579fcddc-7cr2l\" (UID: \"993c1f81-d8b2-46ac-8a86-f6399168de03\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29be73-64579fcddc-7cr2l"
Oct 30 00:43:14.192305 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:14.192274    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ca5363-657fbd45cb-wc7w2" event={"ID":"dd52771e-289e-4bd7-952e-eaddd165eb58","Type":"ContainerStarted","Data":"8b76f8187d6c3fba98c9febdec57a7b546fe3fb19f56316377a0c2d0bbb3ed87"}
Oct 30 00:43:14.192439 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:14.192321    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ca5363-657fbd45cb-wc7w2" event={"ID":"dd52771e-289e-4bd7-952e-eaddd165eb58","Type":"ContainerStarted","Data":"520e946f022b988a79a61dfcb0edceb643e7c6143ff535a8c25dfb296e450cb5"}
Oct 30 00:43:14.192439 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:14.192359    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-ca5363-657fbd45cb-wc7w2"
Oct 30 00:43:14.193947 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:14.193920    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462" event={"ID":"38092938-4c3d-41ae-b7dd-25ddd1869d7d","Type":"ContainerStarted","Data":"03a5de79d6b901a0cac108e4f045620a918a6760d948fc355f2f0dbe681e2a0d"}
Oct 30 00:43:14.194083 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:14.194055    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462"
Oct 30 00:43:14.194899 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:14.194881    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-29be73-58c6d59956-2sszw"
Oct 30 00:43:14.195551 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:14.195526    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca5363-57d48d9d99-ctxwr" event={"ID":"0fa2bddd-6b15-47a8-95cc-8f952594e284","Type":"ContainerStarted","Data":"89ac78482c6e6fa8c27091a9888db9131e252cc331753f2cf702cd4d5c47e195"}
Oct 30 00:43:14.195813 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:14.195708    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca5363-57d48d9d99-ctxwr" event={"ID":"0fa2bddd-6b15-47a8-95cc-8f952594e284","Type":"ContainerStarted","Data":"0b2ed3b08df987477a5538f8c4195463b26622429716dba9ab5b3226ffec3752"}
Oct 30 00:43:14.195955 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:14.195937    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca5363-57d48d9d99-ctxwr"
Oct 30 00:43:14.210853 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:14.210833    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc4tb\" (UniqueName: \"kubernetes.io/projected/993c1f81-d8b2-46ac-8a86-f6399168de03-kube-api-access-pc4tb\") pod \"payload-pre-processing-e2e-worker-w1-29be73-64579fcddc-7cr2l\" (UID: \"993c1f81-d8b2-46ac-8a86-f6399168de03\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29be73-64579fcddc-7cr2l"
Oct 30 00:43:14.233083 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:14.233025    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-ca5363-657fbd45cb-wc7w2" podStartSLOduration=1.233004637 podStartE2EDuration="1.233004637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:14.231087195 +0000 UTC m=+684.618763584" watchObservedRunningTime="2026-10-30 00:43:14.233004637 +0000 UTC m=+684.620681029"
Oct 30 00:43:14.254010 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:14.253984    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29be73-64579fcddc-7cr2l"
Oct 30 00:43:14.274941 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:14.274856    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462" podStartSLOduration=2.990201036 podStartE2EDuration="3.274842111s" totalImagesPullingTime="284.641075ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:14.271629873 +0000 UTC m=+684.659306260" watchObservedRunningTime="2026-10-30 00:43:14.274842111 +0000 UTC m=+684.662518501"
Oct 30 00:43:14.311310 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:14.311257    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca5363-57d48d9d99-ctxwr" podStartSLOduration=2.311242814 podStartE2EDuration="2.311242814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:14.310832034 +0000 UTC m=+684.698508427" watchObservedRunningTime="2026-10-30 00:43:14.311242814 +0000 UTC m=+684.698919203"
Oct 30 00:43:14.493734 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:14.493700    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-29be73-58c6d59956-2sszw"]
Oct 30 00:43:14.495110 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:43:14.495083    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd51483_6bfc_4535_bd66_4c4e74ca70c8.slice/crio-57abb6ac472a9e6ff33eaebc317c13717b9a8ff9b93e3ea45651be2e13b9c7be WatchSource:0}: Error finding container 57abb6ac472a9e6ff33eaebc317c13717b9a8ff9b93e3ea45651be2e13b9c7be: Status 404 returned error can't find the container with id 57abb6ac472a9e6ff33eaebc317c13717b9a8ff9b93e3ea45651be2e13b9c7be
Oct 30 00:43:14.574536 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:43:14.574512    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod993c1f81_d8b2_46ac_8a86_f6399168de03.slice/crio-a3b161b83c062a92cd7fca9ea5af231dab115054705aa1e4e55be8af2fba2978 WatchSource:0}: Error finding container a3b161b83c062a92cd7fca9ea5af231dab115054705aa1e4e55be8af2fba2978: Status 404 returned error can't find the container with id a3b161b83c062a92cd7fca9ea5af231dab115054705aa1e4e55be8af2fba2978
Oct 30 00:43:14.576253 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:14.576230    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-29be73-64579fcddc-7cr2l"]
Oct 30 00:43:15.204275 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:15.204231    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29be73-64579fcddc-7cr2l" event={"ID":"993c1f81-d8b2-46ac-8a86-f6399168de03","Type":"ContainerStarted","Data":"9c84d4f6e1224738e6ca6042606ee0305cb0b739ce43476fa35e74af67074918"}
Oct 30 00:43:15.204750 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:15.204294    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29be73-64579fcddc-7cr2l" event={"ID":"993c1f81-d8b2-46ac-8a86-f6399168de03","Type":"ContainerStarted","Data":"a3b161b83c062a92cd7fca9ea5af231dab115054705aa1e4e55be8af2fba2978"}
Oct 30 00:43:15.204750 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:15.204396    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29be73-64579fcddc-7cr2l"
Oct 30 00:43:15.205891 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:15.205862    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-29be73-58c6d59956-2sszw" event={"ID":"7bd51483-6bfc-4535-bd66-4c4e74ca70c8","Type":"ContainerStarted","Data":"ab90a4a48d48ca894dc293ecd89160bd83c83cdfb92c5de4e7b845a90520eba0"}
Oct 30 00:43:15.206007 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:15.205906    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-29be73-58c6d59956-2sszw" event={"ID":"7bd51483-6bfc-4535-bd66-4c4e74ca70c8","Type":"ContainerStarted","Data":"57abb6ac472a9e6ff33eaebc317c13717b9a8ff9b93e3ea45651be2e13b9c7be"}
Oct 30 00:43:15.206220 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:15.206203    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-29be73-58c6d59956-2sszw"
Oct 30 00:43:15.243758 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:15.243710    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29be73-64579fcddc-7cr2l" podStartSLOduration=2.243698099 podStartE2EDuration="2.243698099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:15.24313328 +0000 UTC m=+685.630809668" watchObservedRunningTime="2026-10-30 00:43:15.243698099 +0000 UTC m=+685.631374488"
Oct 30 00:43:15.282199 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:15.282151    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-29be73-58c6d59956-2sszw" podStartSLOduration=2.282138146 podStartE2EDuration="2.282138146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:15.281362203 +0000 UTC m=+685.669038592" watchObservedRunningTime="2026-10-30 00:43:15.282138146 +0000 UTC m=+685.669814534"
Oct 30 00:43:15.605862 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:15.605762    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq"]
Oct 30 00:43:15.609733 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:15.609712    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq"
Oct 30 00:43:15.627601 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:15.627572    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-ca5363-serving-cert\""
Oct 30 00:43:15.640490 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:15.640465    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq"]
Oct 30 00:43:15.711825 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:15.711791    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb10b912-9abe-4547-83a8-aa63ddb14599-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq\" (UID: \"fb10b912-9abe-4547-83a8-aa63ddb14599\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq"
Oct 30 00:43:15.711985 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:15.711887    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb10b912-9abe-4547-83a8-aa63ddb14599-metrics-certs\") pod \"maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq\" (UID: \"fb10b912-9abe-4547-83a8-aa63ddb14599\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq"
Oct 30 00:43:15.712084 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:15.712060    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wlwc\" (UniqueName: \"kubernetes.io/projected/fb10b912-9abe-4547-83a8-aa63ddb14599-kube-api-access-5wlwc\") pod \"maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq\" (UID: \"fb10b912-9abe-4547-83a8-aa63ddb14599\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq"
Oct 30 00:43:15.712136 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:15.712121    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb10b912-9abe-4547-83a8-aa63ddb14599-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq\" (UID: \"fb10b912-9abe-4547-83a8-aa63ddb14599\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq"
Oct 30 00:43:15.813430 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:15.813390    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wlwc\" (UniqueName: \"kubernetes.io/projected/fb10b912-9abe-4547-83a8-aa63ddb14599-kube-api-access-5wlwc\") pod \"maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq\" (UID: \"fb10b912-9abe-4547-83a8-aa63ddb14599\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq"
Oct 30 00:43:15.813634 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:15.813466    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb10b912-9abe-4547-83a8-aa63ddb14599-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq\" (UID: \"fb10b912-9abe-4547-83a8-aa63ddb14599\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq"
Oct 30 00:43:15.813634 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:15.813543    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb10b912-9abe-4547-83a8-aa63ddb14599-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq\" (UID: \"fb10b912-9abe-4547-83a8-aa63ddb14599\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq"
Oct 30 00:43:15.813634 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:15.813579    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb10b912-9abe-4547-83a8-aa63ddb14599-metrics-certs\") pod \"maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq\" (UID: \"fb10b912-9abe-4547-83a8-aa63ddb14599\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq"
Oct 30 00:43:15.814324 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:15.814254    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb10b912-9abe-4547-83a8-aa63ddb14599-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq\" (UID: \"fb10b912-9abe-4547-83a8-aa63ddb14599\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq"
Oct 30 00:43:15.816120 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:15.816098    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb10b912-9abe-4547-83a8-aa63ddb14599-metrics-certs\") pod \"maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq\" (UID: \"fb10b912-9abe-4547-83a8-aa63ddb14599\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq"
Oct 30 00:43:15.816251 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:15.816230    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb10b912-9abe-4547-83a8-aa63ddb14599-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq\" (UID: \"fb10b912-9abe-4547-83a8-aa63ddb14599\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq"
Oct 30 00:43:15.836152 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:15.836129    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wlwc\" (UniqueName: \"kubernetes.io/projected/fb10b912-9abe-4547-83a8-aa63ddb14599-kube-api-access-5wlwc\") pod \"maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq\" (UID: \"fb10b912-9abe-4547-83a8-aa63ddb14599\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq"
Oct 30 00:43:15.923015 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:15.922979    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq"
Oct 30 00:43:16.224202 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:16.224120    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq"]
Oct 30 00:43:16.229869 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:43:16.229842    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb10b912_9abe_4547_83a8_aa63ddb14599.slice/crio-f17d85b70fb27acbd70372669a6ba00eef25c8c27d81310aa048cdd5ef6a8896 WatchSource:0}: Error finding container f17d85b70fb27acbd70372669a6ba00eef25c8c27d81310aa048cdd5ef6a8896: Status 404 returned error can't find the container with id f17d85b70fb27acbd70372669a6ba00eef25c8c27d81310aa048cdd5ef6a8896
Oct 30 00:43:17.181247 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:17.181218    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp"
Oct 30 00:43:17.191464 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:17.191440    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4"]
Oct 30 00:43:17.195483 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:17.195464    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4"
Oct 30 00:43:17.214020 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:17.213994    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-29be73-serving-cert\""
Oct 30 00:43:17.224551 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:17.223889    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4"]
Oct 30 00:43:17.232408 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:17.232376    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq" event={"ID":"fb10b912-9abe-4547-83a8-aa63ddb14599","Type":"ContainerStarted","Data":"d0daf3efb359bcf377aa5b92bad2d929210896d7582a1c9b11dfc1d8b0df075d"}
Oct 30 00:43:17.232543 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:17.232426    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq" event={"ID":"fb10b912-9abe-4547-83a8-aa63ddb14599","Type":"ContainerStarted","Data":"f17d85b70fb27acbd70372669a6ba00eef25c8c27d81310aa048cdd5ef6a8896"}
Oct 30 00:43:17.328079 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:17.328033    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq" podStartSLOduration=1.994271957 podStartE2EDuration="2.328019351s" totalImagesPullingTime="333.747394ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:17.325704845 +0000 UTC m=+687.713381235" watchObservedRunningTime="2026-10-30 00:43:17.328019351 +0000 UTC m=+687.715695740"
Oct 30 00:43:17.339575 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:17.339548    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4\" (UID: \"44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4"
Oct 30 00:43:17.339711 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:17.339691    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e-metrics-certs\") pod \"maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4\" (UID: \"44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4"
Oct 30 00:43:17.340036 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:17.340009    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e-maas-api-tls\") pod \"maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4\" (UID: \"44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4"
Oct 30 00:43:17.340128 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:17.340076    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76wwd\" (UniqueName: \"kubernetes.io/projected/44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e-kube-api-access-76wwd\") pod \"maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4\" (UID: \"44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4"
Oct 30 00:43:17.445734 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:17.441574    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4\" (UID: \"44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4"
Oct 30 00:43:17.445734 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:17.442381    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4\" (UID: \"44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4"
Oct 30 00:43:17.445734 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:17.442552    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e-metrics-certs\") pod \"maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4\" (UID: \"44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4"
Oct 30 00:43:17.450808 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:17.449337    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e-maas-api-tls\") pod \"maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4\" (UID: \"44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4"
Oct 30 00:43:17.450808 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:17.449534    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76wwd\" (UniqueName: \"kubernetes.io/projected/44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e-kube-api-access-76wwd\") pod \"maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4\" (UID: \"44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4"
Oct 30 00:43:17.453676 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:17.453632    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e-metrics-certs\") pod \"maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4\" (UID: \"44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4"
Oct 30 00:43:17.455977 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:17.455949    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e-maas-api-tls\") pod \"maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4\" (UID: \"44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4"
Oct 30 00:43:17.473536 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:17.473501    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76wwd\" (UniqueName: \"kubernetes.io/projected/44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e-kube-api-access-76wwd\") pod \"maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4\" (UID: \"44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4"
Oct 30 00:43:17.510228 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:17.510200    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4"
Oct 30 00:43:17.811956 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:17.811930    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4"]
Oct 30 00:43:17.813017 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:43:17.812997    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b4eed1_220e_4ff5_a0c2_90a1ccc1d92e.slice/crio-70cb321491fff771712378fca2869d978a169068ec7148f85da8bbc640e776e6 WatchSource:0}: Error finding container 70cb321491fff771712378fca2869d978a169068ec7148f85da8bbc640e776e6: Status 404 returned error can't find the container with id 70cb321491fff771712378fca2869d978a169068ec7148f85da8bbc640e776e6
Oct 30 00:43:18.248509 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:18.248454    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4" event={"ID":"44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e","Type":"ContainerStarted","Data":"5cf540c543d692e86ac2d45b39fdf22dcd7fd71ed7cae3ad9024f26399b588d2"}
Oct 30 00:43:18.250217 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:18.248529    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4" event={"ID":"44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e","Type":"ContainerStarted","Data":"70cb321491fff771712378fca2869d978a169068ec7148f85da8bbc640e776e6"}
Oct 30 00:43:18.250217 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:18.248574    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4"
Oct 30 00:43:18.250217 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:18.248850    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq"
Oct 30 00:43:18.294992 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:18.294941    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4" podStartSLOduration=1.005181043 podStartE2EDuration="1.294927081s" totalImagesPullingTime="289.746038ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:18.292315414 +0000 UTC m=+688.679991804" watchObservedRunningTime="2026-10-30 00:43:18.294927081 +0000 UTC m=+688.682603470"
Oct 30 00:43:19.050876 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:19.050843    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-1a11cf-b489f44f9-lwz7b"]
Oct 30 00:43:19.054978 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:19.054959    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1a11cf-b489f44f9-lwz7b"
Oct 30 00:43:19.073932 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:19.073907    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-1a11cf\""
Oct 30 00:43:19.085615 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:19.085586    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-1a11cf-dockercfg-vj2sk\""
Oct 30 00:43:19.088703 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:19.088675    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-1a11cf-b489f44f9-lwz7b"]
Oct 30 00:43:19.144950 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:19.144917    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs"]
Oct 30 00:43:19.152531 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:19.152501    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs"
Oct 30 00:43:19.170828 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:19.167974    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfgtk\" (UniqueName: \"kubernetes.io/projected/1d8f254c-2610-4cb4-8e28-a59aea8a6de9-kube-api-access-pfgtk\") pod \"payload-pre-processing-e2e-worker-w2-1a11cf-b489f44f9-lwz7b\" (UID: \"1d8f254c-2610-4cb4-8e28-a59aea8a6de9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1a11cf-b489f44f9-lwz7b"
Oct 30 00:43:19.170828 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:19.168031    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d8f254c-2610-4cb4-8e28-a59aea8a6de9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-1a11cf-b489f44f9-lwz7b\" (UID: \"1d8f254c-2610-4cb4-8e28-a59aea8a6de9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1a11cf-b489f44f9-lwz7b"
Oct 30 00:43:19.175386 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:19.175348    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs"]
Oct 30 00:43:19.178820 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:19.178772    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-52650f-serving-cert\""
Oct 30 00:43:19.269409 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:19.269381    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f1a4faf-e09e-4483-a249-c80e24f81129-metrics-certs\") pod \"maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs\" (UID: \"2f1a4faf-e09e-4483-a249-c80e24f81129\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs"
Oct 30 00:43:19.269769 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:19.269495    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1a4faf-e09e-4483-a249-c80e24f81129-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs\" (UID: \"2f1a4faf-e09e-4483-a249-c80e24f81129\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs"
Oct 30 00:43:19.269769 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:19.269528    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkt5f\" (UniqueName: \"kubernetes.io/projected/2f1a4faf-e09e-4483-a249-c80e24f81129-kube-api-access-wkt5f\") pod \"maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs\" (UID: \"2f1a4faf-e09e-4483-a249-c80e24f81129\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs"
Oct 30 00:43:19.269769 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:19.269622    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfgtk\" (UniqueName: \"kubernetes.io/projected/1d8f254c-2610-4cb4-8e28-a59aea8a6de9-kube-api-access-pfgtk\") pod \"payload-pre-processing-e2e-worker-w2-1a11cf-b489f44f9-lwz7b\" (UID: \"1d8f254c-2610-4cb4-8e28-a59aea8a6de9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1a11cf-b489f44f9-lwz7b"
Oct 30 00:43:19.269769 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:19.269650    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d8f254c-2610-4cb4-8e28-a59aea8a6de9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-1a11cf-b489f44f9-lwz7b\" (UID: \"1d8f254c-2610-4cb4-8e28-a59aea8a6de9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1a11cf-b489f44f9-lwz7b"
Oct 30 00:43:19.269769 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:19.269702    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f1a4faf-e09e-4483-a249-c80e24f81129-maas-api-tls\") pod \"maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs\" (UID: \"2f1a4faf-e09e-4483-a249-c80e24f81129\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs"
Oct 30 00:43:19.270371 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:19.270354    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d8f254c-2610-4cb4-8e28-a59aea8a6de9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-1a11cf-b489f44f9-lwz7b\" (UID: \"1d8f254c-2610-4cb4-8e28-a59aea8a6de9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1a11cf-b489f44f9-lwz7b"
Oct 30 00:43:19.294379 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:19.294356    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfgtk\" (UniqueName: \"kubernetes.io/projected/1d8f254c-2610-4cb4-8e28-a59aea8a6de9-kube-api-access-pfgtk\") pod \"payload-pre-processing-e2e-worker-w2-1a11cf-b489f44f9-lwz7b\" (UID: \"1d8f254c-2610-4cb4-8e28-a59aea8a6de9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1a11cf-b489f44f9-lwz7b"
Oct 30 00:43:19.367999 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:19.367974    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1a11cf-b489f44f9-lwz7b"
Oct 30 00:43:19.370484 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:19.370450    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f1a4faf-e09e-4483-a249-c80e24f81129-maas-api-tls\") pod \"maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs\" (UID: \"2f1a4faf-e09e-4483-a249-c80e24f81129\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs"
Oct 30 00:43:19.370619 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:19.370509    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f1a4faf-e09e-4483-a249-c80e24f81129-metrics-certs\") pod \"maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs\" (UID: \"2f1a4faf-e09e-4483-a249-c80e24f81129\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs"
Oct 30 00:43:19.370619 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:43:19.370579    4393 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-52650f-serving-cert: secret "maas-api-e2e-worker-w0-52650f-serving-cert" not found
Oct 30 00:43:19.370735 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:43:19.370653    4393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f1a4faf-e09e-4483-a249-c80e24f81129-maas-api-tls podName:2f1a4faf-e09e-4483-a249-c80e24f81129 nodeName:}" failed. No retries permitted until 2026-10-30 00:43:19.87062488 +0000 UTC m=+690.258301248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2f1a4faf-e09e-4483-a249-c80e24f81129-maas-api-tls") pod "maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs" (UID: "2f1a4faf-e09e-4483-a249-c80e24f81129") : secret "maas-api-e2e-worker-w0-52650f-serving-cert" not found
Oct 30 00:43:19.370840 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:19.370809    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1a4faf-e09e-4483-a249-c80e24f81129-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs\" (UID: \"2f1a4faf-e09e-4483-a249-c80e24f81129\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs"
Oct 30 00:43:19.370899 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:19.370865    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkt5f\" (UniqueName: \"kubernetes.io/projected/2f1a4faf-e09e-4483-a249-c80e24f81129-kube-api-access-wkt5f\") pod \"maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs\" (UID: \"2f1a4faf-e09e-4483-a249-c80e24f81129\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs"
Oct 30 00:43:19.371861 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:19.371526    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1a4faf-e09e-4483-a249-c80e24f81129-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs\" (UID: \"2f1a4faf-e09e-4483-a249-c80e24f81129\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs"
Oct 30 00:43:19.373196 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:19.373170    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f1a4faf-e09e-4483-a249-c80e24f81129-metrics-certs\") pod \"maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs\" (UID: \"2f1a4faf-e09e-4483-a249-c80e24f81129\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs"
Oct 30 00:43:19.395465 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:19.395443    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkt5f\" (UniqueName: \"kubernetes.io/projected/2f1a4faf-e09e-4483-a249-c80e24f81129-kube-api-access-wkt5f\") pod \"maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs\" (UID: \"2f1a4faf-e09e-4483-a249-c80e24f81129\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs"
Oct 30 00:43:19.673121 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:19.673091    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-1a11cf-b489f44f9-lwz7b"]
Oct 30 00:43:19.674352 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:43:19.674323    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d8f254c_2610_4cb4_8e28_a59aea8a6de9.slice/crio-a218151f50a8238c4c22bd7a5a51b7e4b3dacfe9549048e55831c5a8ef8a4d6e WatchSource:0}: Error finding container a218151f50a8238c4c22bd7a5a51b7e4b3dacfe9549048e55831c5a8ef8a4d6e: Status 404 returned error can't find the container with id a218151f50a8238c4c22bd7a5a51b7e4b3dacfe9549048e55831c5a8ef8a4d6e
Oct 30 00:43:19.876121 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:19.876085    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f1a4faf-e09e-4483-a249-c80e24f81129-maas-api-tls\") pod \"maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs\" (UID: \"2f1a4faf-e09e-4483-a249-c80e24f81129\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs"
Oct 30 00:43:19.878497 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:19.878470    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f1a4faf-e09e-4483-a249-c80e24f81129-maas-api-tls\") pod \"maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs\" (UID: \"2f1a4faf-e09e-4483-a249-c80e24f81129\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs"
Oct 30 00:43:20.087452 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:20.087370    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs"
Oct 30 00:43:20.211209 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:20.211179    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462"
Oct 30 00:43:20.263341 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:20.263301    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1a11cf-b489f44f9-lwz7b" event={"ID":"1d8f254c-2610-4cb4-8e28-a59aea8a6de9","Type":"ContainerStarted","Data":"694a819e0ecd39bfb183417f157f995d61297b94a83e1dd353480a0c0a3a3246"}
Oct 30 00:43:20.263523 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:20.263374    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1a11cf-b489f44f9-lwz7b" event={"ID":"1d8f254c-2610-4cb4-8e28-a59aea8a6de9","Type":"ContainerStarted","Data":"a218151f50a8238c4c22bd7a5a51b7e4b3dacfe9549048e55831c5a8ef8a4d6e"}
Oct 30 00:43:20.263523 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:20.263407    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1a11cf-b489f44f9-lwz7b"
Oct 30 00:43:20.309735 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:20.309679    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1a11cf-b489f44f9-lwz7b" podStartSLOduration=1.309665143 podStartE2EDuration="1.309665143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:20.306928997 +0000 UTC m=+690.694605385" watchObservedRunningTime="2026-10-30 00:43:20.309665143 +0000 UTC m=+690.697341532"
Oct 30 00:43:20.399705 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:20.399677    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs"]
Oct 30 00:43:20.399945 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:43:20.399918    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1a4faf_e09e_4483_a249_c80e24f81129.slice/crio-73cfbf7074cd6004387ce62e6cf815effeab8e41eb65116d54f4807392ec3eec WatchSource:0}: Error finding container 73cfbf7074cd6004387ce62e6cf815effeab8e41eb65116d54f4807392ec3eec: Status 404 returned error can't find the container with id 73cfbf7074cd6004387ce62e6cf815effeab8e41eb65116d54f4807392ec3eec
Oct 30 00:43:21.270558 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:21.270522    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs" event={"ID":"2f1a4faf-e09e-4483-a249-c80e24f81129","Type":"ContainerStarted","Data":"6859efafc906d091e92fdd2992064a28fa3db21f8cf94547b6d3014dc7d950f0"}
Oct 30 00:43:21.270733 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:21.270581    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs" event={"ID":"2f1a4faf-e09e-4483-a249-c80e24f81129","Type":"ContainerStarted","Data":"73cfbf7074cd6004387ce62e6cf815effeab8e41eb65116d54f4807392ec3eec"}
Oct 30 00:43:21.270733 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:21.270612    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs"
Oct 30 00:43:21.314252 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:21.314206    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs" podStartSLOduration=1.994628071 podStartE2EDuration="2.314192766s" totalImagesPullingTime="319.564695ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:21.311206979 +0000 UTC m=+691.698883367" watchObservedRunningTime="2026-10-30 00:43:21.314192766 +0000 UTC m=+691.701869154"
Oct 30 00:43:22.179002 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:22.178970    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cefb27-dfdc89d75-jm8hc"
Oct 30 00:43:23.189109 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:23.189083    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-cefb27-88db76d58-8gzgh"
Oct 30 00:43:24.199229 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:24.199199    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-6f4fba-86877b49f8-n59lf"
Oct 30 00:43:24.199623 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:24.199278    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6f4fba-5c479c7dd7-9sxbh"
Oct 30 00:43:24.265884 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:24.265854    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq"
Oct 30 00:43:24.283147 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:24.283111    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4"
Oct 30 00:43:24.414723 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:24.414685    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw"]
Oct 30 00:43:24.419257 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:24.419235    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw"
Oct 30 00:43:24.439397 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:24.437616    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55013a7-d78f-4e6e-bafe-cba27b9462b2-metrics-certs\") pod \"maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw\" (UID: \"e55013a7-d78f-4e6e-bafe-cba27b9462b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw"
Oct 30 00:43:24.439397 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:24.437739    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf884\" (UniqueName: \"kubernetes.io/projected/e55013a7-d78f-4e6e-bafe-cba27b9462b2-kube-api-access-wf884\") pod \"maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw\" (UID: \"e55013a7-d78f-4e6e-bafe-cba27b9462b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw"
Oct 30 00:43:24.439397 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:24.437794    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e55013a7-d78f-4e6e-bafe-cba27b9462b2-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw\" (UID: \"e55013a7-d78f-4e6e-bafe-cba27b9462b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw"
Oct 30 00:43:24.439397 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:24.437821    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e55013a7-d78f-4e6e-bafe-cba27b9462b2-maas-api-tls\") pod \"maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw\" (UID: \"e55013a7-d78f-4e6e-bafe-cba27b9462b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw"
Oct 30 00:43:24.443863 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:24.443822    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw"]
Oct 30 00:43:24.444491 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:24.444468    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-021d6d-serving-cert\""
Oct 30 00:43:24.539391 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:24.539298    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e55013a7-d78f-4e6e-bafe-cba27b9462b2-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw\" (UID: \"e55013a7-d78f-4e6e-bafe-cba27b9462b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw"
Oct 30 00:43:24.539391 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:24.539338    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e55013a7-d78f-4e6e-bafe-cba27b9462b2-maas-api-tls\") pod \"maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw\" (UID: \"e55013a7-d78f-4e6e-bafe-cba27b9462b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw"
Oct 30 00:43:24.539631 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:24.539407    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55013a7-d78f-4e6e-bafe-cba27b9462b2-metrics-certs\") pod \"maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw\" (UID: \"e55013a7-d78f-4e6e-bafe-cba27b9462b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw"
Oct 30 00:43:24.539631 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:24.539487    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf884\" (UniqueName: \"kubernetes.io/projected/e55013a7-d78f-4e6e-bafe-cba27b9462b2-kube-api-access-wf884\") pod \"maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw\" (UID: \"e55013a7-d78f-4e6e-bafe-cba27b9462b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw"
Oct 30 00:43:24.540084 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:24.540060    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e55013a7-d78f-4e6e-bafe-cba27b9462b2-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw\" (UID: \"e55013a7-d78f-4e6e-bafe-cba27b9462b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw"
Oct 30 00:43:24.542076 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:24.542057    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55013a7-d78f-4e6e-bafe-cba27b9462b2-metrics-certs\") pod \"maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw\" (UID: \"e55013a7-d78f-4e6e-bafe-cba27b9462b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw"
Oct 30 00:43:24.542203 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:24.542185    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e55013a7-d78f-4e6e-bafe-cba27b9462b2-maas-api-tls\") pod \"maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw\" (UID: \"e55013a7-d78f-4e6e-bafe-cba27b9462b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw"
Oct 30 00:43:24.562082 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:24.562055    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf884\" (UniqueName: \"kubernetes.io/projected/e55013a7-d78f-4e6e-bafe-cba27b9462b2-kube-api-access-wf884\") pod \"maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw\" (UID: \"e55013a7-d78f-4e6e-bafe-cba27b9462b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw"
Oct 30 00:43:24.735218 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:24.735182    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw"
Oct 30 00:43:25.045006 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:25.044975    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw"]
Oct 30 00:43:25.046117 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:43:25.046089    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode55013a7_d78f_4e6e_bafe_cba27b9462b2.slice/crio-2463f05ab5870d19b8e9b6f2b3c3bc12be58fdbefa2fbd9ec43982adf2b97664 WatchSource:0}: Error finding container 2463f05ab5870d19b8e9b6f2b3c3bc12be58fdbefa2fbd9ec43982adf2b97664: Status 404 returned error can't find the container with id 2463f05ab5870d19b8e9b6f2b3c3bc12be58fdbefa2fbd9ec43982adf2b97664
Oct 30 00:43:25.231485 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:25.231448    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-ca5363-657fbd45cb-wc7w2"
Oct 30 00:43:25.334332 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:25.334230    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw" event={"ID":"e55013a7-d78f-4e6e-bafe-cba27b9462b2","Type":"ContainerStarted","Data":"2463f05ab5870d19b8e9b6f2b3c3bc12be58fdbefa2fbd9ec43982adf2b97664"}
Oct 30 00:43:26.224806 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:26.223684    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca5363-57d48d9d99-ctxwr"
Oct 30 00:43:26.232136 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:26.232112    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29be73-64579fcddc-7cr2l"
Oct 30 00:43:26.350165 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:26.350128    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw" event={"ID":"e55013a7-d78f-4e6e-bafe-cba27b9462b2","Type":"ContainerStarted","Data":"391ee6ab9e492a38f97c1a206aadccb26acb614f48a64f34625f148d87344bb0"}
Oct 30 00:43:26.350365 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:26.350206    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw"
Oct 30 00:43:26.392487 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:26.392438    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw" podStartSLOduration=2.077011942 podStartE2EDuration="2.392422486s" totalImagesPullingTime="315.410544ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:26.390368211 +0000 UTC m=+696.778044601" watchObservedRunningTime="2026-10-30 00:43:26.392422486 +0000 UTC m=+696.780098874"
Oct 30 00:43:26.556076 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:43:26.555984    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:43:26.622278 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:26.622240    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw"]
Oct 30 00:43:26.635024 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:26.634995    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw"
Oct 30 00:43:26.644448 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:26.644421    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw"]
Oct 30 00:43:26.651542 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:26.650054    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-1a11cf-serving-cert\""
Oct 30 00:43:26.667423 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:26.663566    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ea0d2c2-401c-4873-ac44-75481cdfd848-metrics-certs\") pod \"maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw\" (UID: \"4ea0d2c2-401c-4873-ac44-75481cdfd848\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw"
Oct 30 00:43:26.667423 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:26.663622    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ea0d2c2-401c-4873-ac44-75481cdfd848-maas-api-tls\") pod \"maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw\" (UID: \"4ea0d2c2-401c-4873-ac44-75481cdfd848\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw"
Oct 30 00:43:26.667423 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:26.663686    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46d55\" (UniqueName: \"kubernetes.io/projected/4ea0d2c2-401c-4873-ac44-75481cdfd848-kube-api-access-46d55\") pod \"maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw\" (UID: \"4ea0d2c2-401c-4873-ac44-75481cdfd848\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw"
Oct 30 00:43:26.667423 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:26.663811    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea0d2c2-401c-4873-ac44-75481cdfd848-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw\" (UID: \"4ea0d2c2-401c-4873-ac44-75481cdfd848\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw"
Oct 30 00:43:26.765349 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:26.765313    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea0d2c2-401c-4873-ac44-75481cdfd848-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw\" (UID: \"4ea0d2c2-401c-4873-ac44-75481cdfd848\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw"
Oct 30 00:43:26.765549 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:26.765423    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ea0d2c2-401c-4873-ac44-75481cdfd848-metrics-certs\") pod \"maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw\" (UID: \"4ea0d2c2-401c-4873-ac44-75481cdfd848\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw"
Oct 30 00:43:26.765549 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:26.765462    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ea0d2c2-401c-4873-ac44-75481cdfd848-maas-api-tls\") pod \"maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw\" (UID: \"4ea0d2c2-401c-4873-ac44-75481cdfd848\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw"
Oct 30 00:43:26.765549 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:26.765516    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46d55\" (UniqueName: \"kubernetes.io/projected/4ea0d2c2-401c-4873-ac44-75481cdfd848-kube-api-access-46d55\") pod \"maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw\" (UID: \"4ea0d2c2-401c-4873-ac44-75481cdfd848\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw"
Oct 30 00:43:26.766079 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:26.766044    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea0d2c2-401c-4873-ac44-75481cdfd848-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw\" (UID: \"4ea0d2c2-401c-4873-ac44-75481cdfd848\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw"
Oct 30 00:43:26.768263 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:26.768239    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ea0d2c2-401c-4873-ac44-75481cdfd848-metrics-certs\") pod \"maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw\" (UID: \"4ea0d2c2-401c-4873-ac44-75481cdfd848\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw"
Oct 30 00:43:26.768391 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:26.768323    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ea0d2c2-401c-4873-ac44-75481cdfd848-maas-api-tls\") pod \"maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw\" (UID: \"4ea0d2c2-401c-4873-ac44-75481cdfd848\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw"
Oct 30 00:43:26.788423 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:26.788394    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46d55\" (UniqueName: \"kubernetes.io/projected/4ea0d2c2-401c-4873-ac44-75481cdfd848-kube-api-access-46d55\") pod \"maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw\" (UID: \"4ea0d2c2-401c-4873-ac44-75481cdfd848\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw"
Oct 30 00:43:26.957000 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:26.956969    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw"
Oct 30 00:43:27.236216 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:27.236127    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-29be73-58c6d59956-2sszw"
Oct 30 00:43:27.286023 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:27.285999    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs"
Oct 30 00:43:27.287085 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:43:27.287056    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea0d2c2_401c_4873_ac44_75481cdfd848.slice/crio-34906018c61a314c14225b5096f45ea883251df504e410f3f139304c1e5a69e9 WatchSource:0}: Error finding container 34906018c61a314c14225b5096f45ea883251df504e410f3f139304c1e5a69e9: Status 404 returned error can't find the container with id 34906018c61a314c14225b5096f45ea883251df504e410f3f139304c1e5a69e9
Oct 30 00:43:27.291636 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:27.291611    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw"]
Oct 30 00:43:27.360505 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:27.360465    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw" event={"ID":"4ea0d2c2-401c-4873-ac44-75481cdfd848","Type":"ContainerStarted","Data":"34906018c61a314c14225b5096f45ea883251df504e410f3f139304c1e5a69e9"}
Oct 30 00:43:28.366986 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:28.366953    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw" event={"ID":"4ea0d2c2-401c-4873-ac44-75481cdfd848","Type":"ContainerStarted","Data":"4402469f007f85b7fb884e5835052e3291b26eff7ed194fc769b01b5f87d5d9f"}
Oct 30 00:43:28.367362 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:28.367086    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw"
Oct 30 00:43:28.424808 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:28.420940    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw" podStartSLOduration=2.051822066 podStartE2EDuration="2.420919071s" totalImagesPullingTime="369.097005ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:43:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:43:28.409241661 +0000 UTC m=+698.796918051" watchObservedRunningTime="2026-10-30 00:43:28.420919071 +0000 UTC m=+698.808595465"
Oct 30 00:43:31.274222 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:31.274189    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1a11cf-b489f44f9-lwz7b"
Oct 30 00:43:32.366320 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:32.366290    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw"
Oct 30 00:43:34.380302 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:34.380276    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw"
Oct 30 00:43:34.407377 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:34.407348    4393 generic.go:356] "Generic (PLEG): container finished" podID="d4370a67-835f-4d52-97c4-0419593f7975" containerID="7c0f72c0435c5339acec126106d9a6d5cd3b794822b7605714567177c74a3621" exitCode=0
Oct 30 00:43:34.407533 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:34.407419    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x" event={"ID":"d4370a67-835f-4d52-97c4-0419593f7975","Type":"ContainerDied","Data":"7c0f72c0435c5339acec126106d9a6d5cd3b794822b7605714567177c74a3621"}
Oct 30 00:43:34.407831 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:34.407817    4393 scope.go:122] "RemoveContainer" containerID="7c0f72c0435c5339acec126106d9a6d5cd3b794822b7605714567177c74a3621"
Oct 30 00:43:34.409096 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:34.409070    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:43:34.409163 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:34.409135    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:43:34.409199 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:34.409168    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:43:35.434245 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:35.434213    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x" event={"ID":"d4370a67-835f-4d52-97c4-0419593f7975","Type":"ContainerStarted","Data":"8691ab65bf40b8deec2745258bb8493da0eccda6f7bb51ffab2ce426d9a409e9"}
Oct 30 00:43:35.465156 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:35.465126    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:35.466470 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:35.466444    4393 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 00:43:35.466600 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:35.466503    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x" podUID="d4370a67-835f-4d52-97c4-0419593f7975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 00:43:36.464553 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:36.464514    4393 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 00:43:36.465054 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:36.464583    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x" podUID="d4370a67-835f-4d52-97c4-0419593f7975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 00:43:37.126923 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:37.126886    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:37.464915 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:37.464832    4393 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 00:43:37.464915 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:37.464881    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x" podUID="d4370a67-835f-4d52-97c4-0419593f7975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 00:43:38.465472 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:38.465440    4393 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 00:43:38.465917 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:38.465489    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x" podUID="d4370a67-835f-4d52-97c4-0419593f7975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 00:43:39.465204 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:39.465168    4393 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 00:43:39.465400 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:39.465230    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x" podUID="d4370a67-835f-4d52-97c4-0419593f7975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 00:43:40.465006 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:40.464974    4393 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 00:43:40.465523 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:40.465022    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x" podUID="d4370a67-835f-4d52-97c4-0419593f7975" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 00:43:41.470010 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:41.469974    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:41.471358 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:43:41.471336    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 00:43:56.510333 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:43:56.510294    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:44:26.509943 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:44:26.509905    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:44:49.892934 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:49.892904    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"]
Oct 30 00:44:49.896030 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:49.896013    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"
Oct 30 00:44:49.910422 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:49.910397    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-29be73-openshift-default-dockercfg-xqn9v\""
Oct 30 00:44:49.932982 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:49.932954    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"]
Oct 30 00:44:50.045267 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:50.045232    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/141a2288-b7f5-46e6-85cb-c77e432999d3-istiod-ca-cert\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h\" (UID: \"141a2288-b7f5-46e6-85cb-c77e432999d3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"
Oct 30 00:44:50.045464 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:50.045283    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/141a2288-b7f5-46e6-85cb-c77e432999d3-workload-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h\" (UID: \"141a2288-b7f5-46e6-85cb-c77e432999d3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"
Oct 30 00:44:50.045464 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:50.045432    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmtdn\" (UniqueName: \"kubernetes.io/projected/141a2288-b7f5-46e6-85cb-c77e432999d3-kube-api-access-xmtdn\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h\" (UID: \"141a2288-b7f5-46e6-85cb-c77e432999d3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"
Oct 30 00:44:50.045590 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:50.045468    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/141a2288-b7f5-46e6-85cb-c77e432999d3-istio-envoy\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h\" (UID: \"141a2288-b7f5-46e6-85cb-c77e432999d3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"
Oct 30 00:44:50.045590 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:50.045544    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/141a2288-b7f5-46e6-85cb-c77e432999d3-istio-data\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h\" (UID: \"141a2288-b7f5-46e6-85cb-c77e432999d3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"
Oct 30 00:44:50.045700 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:50.045665    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/141a2288-b7f5-46e6-85cb-c77e432999d3-istio-token\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h\" (UID: \"141a2288-b7f5-46e6-85cb-c77e432999d3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"
Oct 30 00:44:50.045757 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:50.045718    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/141a2288-b7f5-46e6-85cb-c77e432999d3-credential-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h\" (UID: \"141a2288-b7f5-46e6-85cb-c77e432999d3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"
Oct 30 00:44:50.045757 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:50.045744    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/141a2288-b7f5-46e6-85cb-c77e432999d3-istio-podinfo\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h\" (UID: \"141a2288-b7f5-46e6-85cb-c77e432999d3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"
Oct 30 00:44:50.045939 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:50.045914    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/141a2288-b7f5-46e6-85cb-c77e432999d3-workload-certs\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h\" (UID: \"141a2288-b7f5-46e6-85cb-c77e432999d3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"
Oct 30 00:44:50.147247 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:50.147170    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmtdn\" (UniqueName: \"kubernetes.io/projected/141a2288-b7f5-46e6-85cb-c77e432999d3-kube-api-access-xmtdn\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h\" (UID: \"141a2288-b7f5-46e6-85cb-c77e432999d3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"
Oct 30 00:44:50.147247 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:50.147207    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/141a2288-b7f5-46e6-85cb-c77e432999d3-istio-envoy\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h\" (UID: \"141a2288-b7f5-46e6-85cb-c77e432999d3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"
Oct 30 00:44:50.147247 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:50.147236    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/141a2288-b7f5-46e6-85cb-c77e432999d3-istio-data\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h\" (UID: \"141a2288-b7f5-46e6-85cb-c77e432999d3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"
Oct 30 00:44:50.147485 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:50.147272    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/141a2288-b7f5-46e6-85cb-c77e432999d3-istio-token\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h\" (UID: \"141a2288-b7f5-46e6-85cb-c77e432999d3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"
Oct 30 00:44:50.147485 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:50.147302    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/141a2288-b7f5-46e6-85cb-c77e432999d3-credential-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h\" (UID: \"141a2288-b7f5-46e6-85cb-c77e432999d3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"
Oct 30 00:44:50.147485 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:50.147318    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/141a2288-b7f5-46e6-85cb-c77e432999d3-istio-podinfo\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h\" (UID: \"141a2288-b7f5-46e6-85cb-c77e432999d3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"
Oct 30 00:44:50.147637 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:50.147492    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/141a2288-b7f5-46e6-85cb-c77e432999d3-workload-certs\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h\" (UID: \"141a2288-b7f5-46e6-85cb-c77e432999d3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"
Oct 30 00:44:50.147637 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:50.147626    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/141a2288-b7f5-46e6-85cb-c77e432999d3-istiod-ca-cert\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h\" (UID: \"141a2288-b7f5-46e6-85cb-c77e432999d3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"
Oct 30 00:44:50.147946 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:50.147657    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/141a2288-b7f5-46e6-85cb-c77e432999d3-workload-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h\" (UID: \"141a2288-b7f5-46e6-85cb-c77e432999d3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"
Oct 30 00:44:50.147946 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:50.147721    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/141a2288-b7f5-46e6-85cb-c77e432999d3-istio-data\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h\" (UID: \"141a2288-b7f5-46e6-85cb-c77e432999d3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"
Oct 30 00:44:50.147946 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:50.147733    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/141a2288-b7f5-46e6-85cb-c77e432999d3-credential-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h\" (UID: \"141a2288-b7f5-46e6-85cb-c77e432999d3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"
Oct 30 00:44:50.147946 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:50.147897    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/141a2288-b7f5-46e6-85cb-c77e432999d3-workload-certs\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h\" (UID: \"141a2288-b7f5-46e6-85cb-c77e432999d3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"
Oct 30 00:44:50.148130 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:50.148000    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/141a2288-b7f5-46e6-85cb-c77e432999d3-workload-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h\" (UID: \"141a2288-b7f5-46e6-85cb-c77e432999d3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"
Oct 30 00:44:50.148626 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:50.148596    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/141a2288-b7f5-46e6-85cb-c77e432999d3-istiod-ca-cert\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h\" (UID: \"141a2288-b7f5-46e6-85cb-c77e432999d3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"
Oct 30 00:44:50.149699 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:50.149676    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/141a2288-b7f5-46e6-85cb-c77e432999d3-istio-envoy\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h\" (UID: \"141a2288-b7f5-46e6-85cb-c77e432999d3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"
Oct 30 00:44:50.150009 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:50.149993    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/141a2288-b7f5-46e6-85cb-c77e432999d3-istio-podinfo\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h\" (UID: \"141a2288-b7f5-46e6-85cb-c77e432999d3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"
Oct 30 00:44:50.169839 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:50.169817    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/141a2288-b7f5-46e6-85cb-c77e432999d3-istio-token\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h\" (UID: \"141a2288-b7f5-46e6-85cb-c77e432999d3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"
Oct 30 00:44:50.170001 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:50.169983    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmtdn\" (UniqueName: \"kubernetes.io/projected/141a2288-b7f5-46e6-85cb-c77e432999d3-kube-api-access-xmtdn\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h\" (UID: \"141a2288-b7f5-46e6-85cb-c77e432999d3\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"
Oct 30 00:44:50.207655 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:50.207625    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"
Oct 30 00:44:50.527565 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:50.527541    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"]
Oct 30 00:44:50.528380 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:44:50.528354    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod141a2288_b7f5_46e6_85cb_c77e432999d3.slice/crio-c5e3f4758afa9e58e21ba9a1f27cbd035c44ffd511c132c34a78e953468dc8c8 WatchSource:0}: Error finding container c5e3f4758afa9e58e21ba9a1f27cbd035c44ffd511c132c34a78e953468dc8c8: Status 404 returned error can't find the container with id c5e3f4758afa9e58e21ba9a1f27cbd035c44ffd511c132c34a78e953468dc8c8
Oct 30 00:44:50.530642 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:50.530600    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:44:50.530721 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:50.530668    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:44:50.530721 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:50.530698    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:44:50.967431 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:50.967394    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h" event={"ID":"141a2288-b7f5-46e6-85cb-c77e432999d3","Type":"ContainerStarted","Data":"10249610d99c7036916c02e0ffff104f677ffbf06619e8b413c1223498bcd08f"}
Oct 30 00:44:50.969153 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:50.969123    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h" event={"ID":"141a2288-b7f5-46e6-85cb-c77e432999d3","Type":"ContainerStarted","Data":"c5e3f4758afa9e58e21ba9a1f27cbd035c44ffd511c132c34a78e953468dc8c8"}
Oct 30 00:44:51.046171 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:51.046113    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h" podStartSLOduration=2.046095063 podStartE2EDuration="2.046095063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:44:51.033003047 +0000 UTC m=+781.420679436" watchObservedRunningTime="2026-10-30 00:44:51.046095063 +0000 UTC m=+781.433771456"
Oct 30 00:44:51.227086 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:51.226988    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"
Oct 30 00:44:51.229030 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:51.229001    4393 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 00:44:51.229137 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:51.229075    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h" podUID="141a2288-b7f5-46e6-85cb-c77e432999d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 00:44:52.217605 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:52.217146    4393 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 00:44:52.217605 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:52.217239    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h" podUID="141a2288-b7f5-46e6-85cb-c77e432999d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 00:44:53.208148 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:53.208112    4393 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 00:44:53.208432 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:53.208402    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h" podUID="141a2288-b7f5-46e6-85cb-c77e432999d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 00:44:54.214803 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:54.208100    4393 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 00:44:54.214803 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:54.208158    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h" podUID="141a2288-b7f5-46e6-85cb-c77e432999d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 00:44:55.227808 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:55.227402    4393 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 00:44:55.227808 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:55.227473    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h" podUID="141a2288-b7f5-46e6-85cb-c77e432999d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 00:44:56.225944 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:56.225906    4393 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 00:44:56.226884 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:56.226838    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h" podUID="141a2288-b7f5-46e6-85cb-c77e432999d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 00:44:56.510595 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:44:56.510502    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:44:57.331809 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:57.331757    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"
Oct 30 00:44:57.332182 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:57.332158    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"
Oct 30 00:44:57.333228 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:44:57.333204    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"
Oct 30 00:45:20.369184 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:20.369154    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"]
Oct 30 00:45:20.374902 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:20.374881    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"
Oct 30 00:45:20.403560 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:20.403530    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-021d6d-openshift-default-dockercfg-trf72\""
Oct 30 00:45:20.408930 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:20.408896    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"]
Oct 30 00:45:20.447039 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:20.446999    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d42228bf-b9c4-4be5-bc94-ae937377708b-istio-token\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx\" (UID: \"d42228bf-b9c4-4be5-bc94-ae937377708b\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"
Oct 30 00:45:20.447224 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:20.447045    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42228bf-b9c4-4be5-bc94-ae937377708b-workload-certs\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx\" (UID: \"d42228bf-b9c4-4be5-bc94-ae937377708b\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"
Oct 30 00:45:20.447224 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:20.447141    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfkzm\" (UniqueName: \"kubernetes.io/projected/d42228bf-b9c4-4be5-bc94-ae937377708b-kube-api-access-bfkzm\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx\" (UID: \"d42228bf-b9c4-4be5-bc94-ae937377708b\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"
Oct 30 00:45:20.447224 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:20.447172    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42228bf-b9c4-4be5-bc94-ae937377708b-istio-data\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx\" (UID: \"d42228bf-b9c4-4be5-bc94-ae937377708b\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"
Oct 30 00:45:20.447224 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:20.447190    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42228bf-b9c4-4be5-bc94-ae937377708b-workload-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx\" (UID: \"d42228bf-b9c4-4be5-bc94-ae937377708b\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"
Oct 30 00:45:20.447461 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:20.447266    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42228bf-b9c4-4be5-bc94-ae937377708b-istiod-ca-cert\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx\" (UID: \"d42228bf-b9c4-4be5-bc94-ae937377708b\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"
Oct 30 00:45:20.447461 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:20.447295    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42228bf-b9c4-4be5-bc94-ae937377708b-credential-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx\" (UID: \"d42228bf-b9c4-4be5-bc94-ae937377708b\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"
Oct 30 00:45:20.447461 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:20.447314    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42228bf-b9c4-4be5-bc94-ae937377708b-istio-podinfo\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx\" (UID: \"d42228bf-b9c4-4be5-bc94-ae937377708b\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"
Oct 30 00:45:20.447461 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:20.447347    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42228bf-b9c4-4be5-bc94-ae937377708b-istio-envoy\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx\" (UID: \"d42228bf-b9c4-4be5-bc94-ae937377708b\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"
Oct 30 00:45:20.548520 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:20.548487    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d42228bf-b9c4-4be5-bc94-ae937377708b-istio-token\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx\" (UID: \"d42228bf-b9c4-4be5-bc94-ae937377708b\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"
Oct 30 00:45:20.548699 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:20.548539    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42228bf-b9c4-4be5-bc94-ae937377708b-workload-certs\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx\" (UID: \"d42228bf-b9c4-4be5-bc94-ae937377708b\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"
Oct 30 00:45:20.548699 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:20.548605    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfkzm\" (UniqueName: \"kubernetes.io/projected/d42228bf-b9c4-4be5-bc94-ae937377708b-kube-api-access-bfkzm\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx\" (UID: \"d42228bf-b9c4-4be5-bc94-ae937377708b\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"
Oct 30 00:45:20.548699 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:20.548637    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42228bf-b9c4-4be5-bc94-ae937377708b-istio-data\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx\" (UID: \"d42228bf-b9c4-4be5-bc94-ae937377708b\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"
Oct 30 00:45:20.548699 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:20.548664    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42228bf-b9c4-4be5-bc94-ae937377708b-workload-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx\" (UID: \"d42228bf-b9c4-4be5-bc94-ae937377708b\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"
Oct 30 00:45:20.548941 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:20.548701    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42228bf-b9c4-4be5-bc94-ae937377708b-istiod-ca-cert\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx\" (UID: \"d42228bf-b9c4-4be5-bc94-ae937377708b\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"
Oct 30 00:45:20.548941 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:20.548728    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42228bf-b9c4-4be5-bc94-ae937377708b-credential-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx\" (UID: \"d42228bf-b9c4-4be5-bc94-ae937377708b\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"
Oct 30 00:45:20.548941 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:20.548753    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42228bf-b9c4-4be5-bc94-ae937377708b-istio-podinfo\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx\" (UID: \"d42228bf-b9c4-4be5-bc94-ae937377708b\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"
Oct 30 00:45:20.548941 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:20.548815    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42228bf-b9c4-4be5-bc94-ae937377708b-istio-envoy\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx\" (UID: \"d42228bf-b9c4-4be5-bc94-ae937377708b\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"
Oct 30 00:45:20.549142 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:20.549072    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42228bf-b9c4-4be5-bc94-ae937377708b-workload-certs\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx\" (UID: \"d42228bf-b9c4-4be5-bc94-ae937377708b\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"
Oct 30 00:45:20.549292 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:20.549266    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42228bf-b9c4-4be5-bc94-ae937377708b-istio-data\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx\" (UID: \"d42228bf-b9c4-4be5-bc94-ae937377708b\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"
Oct 30 00:45:20.549369 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:20.549287    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42228bf-b9c4-4be5-bc94-ae937377708b-workload-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx\" (UID: \"d42228bf-b9c4-4be5-bc94-ae937377708b\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"
Oct 30 00:45:20.549430 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:20.549408    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42228bf-b9c4-4be5-bc94-ae937377708b-credential-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx\" (UID: \"d42228bf-b9c4-4be5-bc94-ae937377708b\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"
Oct 30 00:45:20.549637 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:20.549618    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42228bf-b9c4-4be5-bc94-ae937377708b-istiod-ca-cert\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx\" (UID: \"d42228bf-b9c4-4be5-bc94-ae937377708b\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"
Oct 30 00:45:20.551368 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:20.551348    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42228bf-b9c4-4be5-bc94-ae937377708b-istio-envoy\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx\" (UID: \"d42228bf-b9c4-4be5-bc94-ae937377708b\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"
Oct 30 00:45:20.551512 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:20.551494    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42228bf-b9c4-4be5-bc94-ae937377708b-istio-podinfo\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx\" (UID: \"d42228bf-b9c4-4be5-bc94-ae937377708b\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"
Oct 30 00:45:20.571314 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:20.571279    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d42228bf-b9c4-4be5-bc94-ae937377708b-istio-token\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx\" (UID: \"d42228bf-b9c4-4be5-bc94-ae937377708b\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"
Oct 30 00:45:20.571409 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:20.571335    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfkzm\" (UniqueName: \"kubernetes.io/projected/d42228bf-b9c4-4be5-bc94-ae937377708b-kube-api-access-bfkzm\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx\" (UID: \"d42228bf-b9c4-4be5-bc94-ae937377708b\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"
Oct 30 00:45:20.687318 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:20.687229    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"
Oct 30 00:45:21.042990 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:21.042955    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"]
Oct 30 00:45:21.044376 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:45:21.044346    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd42228bf_b9c4_4be5_bc94_ae937377708b.slice/crio-651b07a4ccd0b9bba2f7a415c6e2fea838d572b933f746916da069514a7c5199 WatchSource:0}: Error finding container 651b07a4ccd0b9bba2f7a415c6e2fea838d572b933f746916da069514a7c5199: Status 404 returned error can't find the container with id 651b07a4ccd0b9bba2f7a415c6e2fea838d572b933f746916da069514a7c5199
Oct 30 00:45:21.047190 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:21.047157    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:45:21.047276 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:21.047233    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:45:21.047317 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:21.047279    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:45:21.161315 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:21.161277    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx" event={"ID":"d42228bf-b9c4-4be5-bc94-ae937377708b","Type":"ContainerStarted","Data":"a3961aa54255c8969a872cc6c2795415c4ab2a2d6b034c9eb4f2b3f6bbda05e6"}
Oct 30 00:45:21.161484 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:21.161340    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx" event={"ID":"d42228bf-b9c4-4be5-bc94-ae937377708b","Type":"ContainerStarted","Data":"651b07a4ccd0b9bba2f7a415c6e2fea838d572b933f746916da069514a7c5199"}
Oct 30 00:45:21.211137 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:21.211081    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx" podStartSLOduration=1.211065214 podStartE2EDuration="1.211065214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:45:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:45:21.20916857 +0000 UTC m=+811.596844949" watchObservedRunningTime="2026-10-30 00:45:21.211065214 +0000 UTC m=+811.598741603"
Oct 30 00:45:21.687613 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:21.687571    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"
Oct 30 00:45:22.689738 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:22.689705    4393 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 00:45:22.690122 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:22.689766    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx" podUID="d42228bf-b9c4-4be5-bc94-ae937377708b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 00:45:23.687965 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:23.687927    4393 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 00:45:23.688122 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:23.687989    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx" podUID="d42228bf-b9c4-4be5-bc94-ae937377708b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 00:45:24.688112 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:24.688072    4393 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 00:45:24.688510 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:24.688144    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx" podUID="d42228bf-b9c4-4be5-bc94-ae937377708b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 00:45:25.687978 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:25.687948    4393 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 00:45:25.688226 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:25.688207    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx" podUID="d42228bf-b9c4-4be5-bc94-ae937377708b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 00:45:26.547806 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:45:26.547629    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:45:26.688132 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:26.688085    4393 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 00:45:26.688342 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:26.688145    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx" podUID="d42228bf-b9c4-4be5-bc94-ae937377708b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 00:45:27.688260 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:27.688225    4393 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 00:45:27.688449 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:27.688293    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx" podUID="d42228bf-b9c4-4be5-bc94-ae937377708b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 00:45:28.990220 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:28.990185    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"
Oct 30 00:45:28.990681 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:28.990546    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"
Oct 30 00:45:29.002504 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:29.002481    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"
Oct 30 00:45:34.920066 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:34.920025    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"]
Oct 30 00:45:34.923097 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:34.923080    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"
Oct 30 00:45:34.938079 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:34.938051    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-6f4fba-openshift-default-dockercfg-l52jt\""
Oct 30 00:45:34.953903 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:34.953875    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"]
Oct 30 00:45:34.981518 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:34.981486    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7afd2398-4294-4236-95d4-82b86a99ca43-istiod-ca-cert\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9\" (UID: \"7afd2398-4294-4236-95d4-82b86a99ca43\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"
Oct 30 00:45:34.981692 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:34.981642    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7afd2398-4294-4236-95d4-82b86a99ca43-istio-token\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9\" (UID: \"7afd2398-4294-4236-95d4-82b86a99ca43\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"
Oct 30 00:45:34.981753 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:34.981690    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7afd2398-4294-4236-95d4-82b86a99ca43-credential-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9\" (UID: \"7afd2398-4294-4236-95d4-82b86a99ca43\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"
Oct 30 00:45:34.981843 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:34.981821    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5z7v\" (UniqueName: \"kubernetes.io/projected/7afd2398-4294-4236-95d4-82b86a99ca43-kube-api-access-h5z7v\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9\" (UID: \"7afd2398-4294-4236-95d4-82b86a99ca43\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"
Oct 30 00:45:34.981962 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:34.981943    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7afd2398-4294-4236-95d4-82b86a99ca43-istio-data\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9\" (UID: \"7afd2398-4294-4236-95d4-82b86a99ca43\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"
Oct 30 00:45:34.982000 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:34.981990    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7afd2398-4294-4236-95d4-82b86a99ca43-istio-podinfo\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9\" (UID: \"7afd2398-4294-4236-95d4-82b86a99ca43\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"
Oct 30 00:45:34.982050 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:34.982036    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7afd2398-4294-4236-95d4-82b86a99ca43-workload-certs\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9\" (UID: \"7afd2398-4294-4236-95d4-82b86a99ca43\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"
Oct 30 00:45:34.982153 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:34.982137    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7afd2398-4294-4236-95d4-82b86a99ca43-workload-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9\" (UID: \"7afd2398-4294-4236-95d4-82b86a99ca43\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"
Oct 30 00:45:34.982256 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:34.982171    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7afd2398-4294-4236-95d4-82b86a99ca43-istio-envoy\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9\" (UID: \"7afd2398-4294-4236-95d4-82b86a99ca43\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"
Oct 30 00:45:35.083333 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.083297    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7afd2398-4294-4236-95d4-82b86a99ca43-istio-token\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9\" (UID: \"7afd2398-4294-4236-95d4-82b86a99ca43\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"
Oct 30 00:45:35.083333 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.083338    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7afd2398-4294-4236-95d4-82b86a99ca43-credential-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9\" (UID: \"7afd2398-4294-4236-95d4-82b86a99ca43\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"
Oct 30 00:45:35.083540 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.083381    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5z7v\" (UniqueName: \"kubernetes.io/projected/7afd2398-4294-4236-95d4-82b86a99ca43-kube-api-access-h5z7v\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9\" (UID: \"7afd2398-4294-4236-95d4-82b86a99ca43\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"
Oct 30 00:45:35.083540 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.083466    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7afd2398-4294-4236-95d4-82b86a99ca43-istio-data\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9\" (UID: \"7afd2398-4294-4236-95d4-82b86a99ca43\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"
Oct 30 00:45:35.083540 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.083501    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7afd2398-4294-4236-95d4-82b86a99ca43-istio-podinfo\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9\" (UID: \"7afd2398-4294-4236-95d4-82b86a99ca43\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"
Oct 30 00:45:35.083699 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.083545    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7afd2398-4294-4236-95d4-82b86a99ca43-workload-certs\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9\" (UID: \"7afd2398-4294-4236-95d4-82b86a99ca43\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"
Oct 30 00:45:35.083774 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.083744    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7afd2398-4294-4236-95d4-82b86a99ca43-workload-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9\" (UID: \"7afd2398-4294-4236-95d4-82b86a99ca43\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"
Oct 30 00:45:35.083878 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.083831    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7afd2398-4294-4236-95d4-82b86a99ca43-istio-envoy\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9\" (UID: \"7afd2398-4294-4236-95d4-82b86a99ca43\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"
Oct 30 00:45:35.083878 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.083868    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7afd2398-4294-4236-95d4-82b86a99ca43-istiod-ca-cert\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9\" (UID: \"7afd2398-4294-4236-95d4-82b86a99ca43\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"
Oct 30 00:45:35.084041 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.083928    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7afd2398-4294-4236-95d4-82b86a99ca43-credential-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9\" (UID: \"7afd2398-4294-4236-95d4-82b86a99ca43\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"
Oct 30 00:45:35.084041 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.083966    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7afd2398-4294-4236-95d4-82b86a99ca43-istio-data\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9\" (UID: \"7afd2398-4294-4236-95d4-82b86a99ca43\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"
Oct 30 00:45:35.084210 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.084180    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7afd2398-4294-4236-95d4-82b86a99ca43-workload-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9\" (UID: \"7afd2398-4294-4236-95d4-82b86a99ca43\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"
Oct 30 00:45:35.084522 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.084491    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7afd2398-4294-4236-95d4-82b86a99ca43-istiod-ca-cert\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9\" (UID: \"7afd2398-4294-4236-95d4-82b86a99ca43\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"
Oct 30 00:45:35.084640 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.084616    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7afd2398-4294-4236-95d4-82b86a99ca43-workload-certs\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9\" (UID: \"7afd2398-4294-4236-95d4-82b86a99ca43\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"
Oct 30 00:45:35.086024 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.085995    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7afd2398-4294-4236-95d4-82b86a99ca43-istio-envoy\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9\" (UID: \"7afd2398-4294-4236-95d4-82b86a99ca43\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"
Oct 30 00:45:35.086229 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.086206    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7afd2398-4294-4236-95d4-82b86a99ca43-istio-podinfo\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9\" (UID: \"7afd2398-4294-4236-95d4-82b86a99ca43\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"
Oct 30 00:45:35.106298 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.106276    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7afd2398-4294-4236-95d4-82b86a99ca43-istio-token\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9\" (UID: \"7afd2398-4294-4236-95d4-82b86a99ca43\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"
Oct 30 00:45:35.106501 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.106480    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5z7v\" (UniqueName: \"kubernetes.io/projected/7afd2398-4294-4236-95d4-82b86a99ca43-kube-api-access-h5z7v\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9\" (UID: \"7afd2398-4294-4236-95d4-82b86a99ca43\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"
Oct 30 00:45:35.167660 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.167624    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"]
Oct 30 00:45:35.171030 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.170968    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"
Oct 30 00:45:35.190956 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.190576    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-ca5363-openshift-default-dockercfg-m4tkh\""
Oct 30 00:45:35.195628 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.195590    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"]
Oct 30 00:45:35.236039 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.236016    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"
Oct 30 00:45:35.287670 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.286276    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-istio-data\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg\" (UID: \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"
Oct 30 00:45:35.287670 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.286330    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-istiod-ca-cert\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg\" (UID: \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"
Oct 30 00:45:35.287670 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.286432    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-workload-certs\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg\" (UID: \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"
Oct 30 00:45:35.287670 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.286460    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-istio-envoy\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg\" (UID: \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"
Oct 30 00:45:35.287670 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.286491    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cq5z\" (UniqueName: \"kubernetes.io/projected/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-kube-api-access-6cq5z\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg\" (UID: \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"
Oct 30 00:45:35.287670 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.286589    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-credential-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg\" (UID: \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"
Oct 30 00:45:35.287670 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.286749    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-istio-token\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg\" (UID: \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"
Oct 30 00:45:35.287670 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.286773    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-workload-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg\" (UID: \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"
Oct 30 00:45:35.287670 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.286859    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-istio-podinfo\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg\" (UID: \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"
Oct 30 00:45:35.388519 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.388480    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-istio-data\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg\" (UID: \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"
Oct 30 00:45:35.388519 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.388525    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-istiod-ca-cert\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg\" (UID: \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"
Oct 30 00:45:35.388734 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.388565    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-workload-certs\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg\" (UID: \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"
Oct 30 00:45:35.388734 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.388590    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-istio-envoy\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg\" (UID: \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"
Oct 30 00:45:35.388734 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.388621    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cq5z\" (UniqueName: \"kubernetes.io/projected/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-kube-api-access-6cq5z\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg\" (UID: \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"
Oct 30 00:45:35.388734 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.388654    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-credential-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg\" (UID: \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"
Oct 30 00:45:35.388984 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.388838    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-istio-token\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg\" (UID: \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"
Oct 30 00:45:35.388984 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.388873    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-workload-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg\" (UID: \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"
Oct 30 00:45:35.388984 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.388900    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-istio-podinfo\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg\" (UID: \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"
Oct 30 00:45:35.389141 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.389005    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-workload-certs\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg\" (UID: \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"
Oct 30 00:45:35.389141 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.389019    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-istio-data\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg\" (UID: \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"
Oct 30 00:45:35.389141 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.389083    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-credential-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg\" (UID: \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"
Oct 30 00:45:35.389354 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.389332    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-workload-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg\" (UID: \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"
Oct 30 00:45:35.389416 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.389388    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-istiod-ca-cert\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg\" (UID: \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"
Oct 30 00:45:35.391410 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.391390    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-istio-envoy\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg\" (UID: \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"
Oct 30 00:45:35.391633 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.391605    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-istio-podinfo\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg\" (UID: \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"
Oct 30 00:45:35.411508 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.411485    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-istio-token\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg\" (UID: \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"
Oct 30 00:45:35.411928 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.411908    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cq5z\" (UniqueName: \"kubernetes.io/projected/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-kube-api-access-6cq5z\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg\" (UID: \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"
Oct 30 00:45:35.484069 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.483993    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"
Oct 30 00:45:35.504106 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.504079    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"]
Oct 30 00:45:35.507895 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.507872    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"
Oct 30 00:45:35.523022 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.522987    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-1a11cf-openshift-default-dockercfg-j69mc\""
Oct 30 00:45:35.540838 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.538106    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"]
Oct 30 00:45:35.565442 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.565370    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"]
Oct 30 00:45:35.569170 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.569126    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:45:35.569282 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.569209    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:45:35.569282 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.569237    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:45:35.591682 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.591653    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75776f35-e39d-4823-ac2a-61e5b4e69619-credential-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz\" (UID: \"75776f35-e39d-4823-ac2a-61e5b4e69619\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"
Oct 30 00:45:35.591852 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.591756    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59s52\" (UniqueName: \"kubernetes.io/projected/75776f35-e39d-4823-ac2a-61e5b4e69619-kube-api-access-59s52\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz\" (UID: \"75776f35-e39d-4823-ac2a-61e5b4e69619\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"
Oct 30 00:45:35.591852 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.591813    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75776f35-e39d-4823-ac2a-61e5b4e69619-workload-certs\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz\" (UID: \"75776f35-e39d-4823-ac2a-61e5b4e69619\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"
Oct 30 00:45:35.592063 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.591913    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75776f35-e39d-4823-ac2a-61e5b4e69619-workload-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz\" (UID: \"75776f35-e39d-4823-ac2a-61e5b4e69619\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"
Oct 30 00:45:35.592063 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.591944    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75776f35-e39d-4823-ac2a-61e5b4e69619-istio-envoy\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz\" (UID: \"75776f35-e39d-4823-ac2a-61e5b4e69619\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"
Oct 30 00:45:35.592063 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.592011    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75776f35-e39d-4823-ac2a-61e5b4e69619-istio-podinfo\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz\" (UID: \"75776f35-e39d-4823-ac2a-61e5b4e69619\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"
Oct 30 00:45:35.592063 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.592052    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75776f35-e39d-4823-ac2a-61e5b4e69619-istio-data\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz\" (UID: \"75776f35-e39d-4823-ac2a-61e5b4e69619\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"
Oct 30 00:45:35.592281 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.592087    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75776f35-e39d-4823-ac2a-61e5b4e69619-istio-token\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz\" (UID: \"75776f35-e39d-4823-ac2a-61e5b4e69619\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"
Oct 30 00:45:35.592340 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.592278    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75776f35-e39d-4823-ac2a-61e5b4e69619-istiod-ca-cert\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz\" (UID: \"75776f35-e39d-4823-ac2a-61e5b4e69619\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"
Oct 30 00:45:35.693719 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.693684    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75776f35-e39d-4823-ac2a-61e5b4e69619-workload-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz\" (UID: \"75776f35-e39d-4823-ac2a-61e5b4e69619\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"
Oct 30 00:45:35.693900 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.693738    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75776f35-e39d-4823-ac2a-61e5b4e69619-istio-envoy\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz\" (UID: \"75776f35-e39d-4823-ac2a-61e5b4e69619\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"
Oct 30 00:45:35.693900 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.693839    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75776f35-e39d-4823-ac2a-61e5b4e69619-istio-podinfo\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz\" (UID: \"75776f35-e39d-4823-ac2a-61e5b4e69619\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"
Oct 30 00:45:35.693900 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.693884    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75776f35-e39d-4823-ac2a-61e5b4e69619-istio-data\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz\" (UID: \"75776f35-e39d-4823-ac2a-61e5b4e69619\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"
Oct 30 00:45:35.694053 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.693917    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75776f35-e39d-4823-ac2a-61e5b4e69619-istio-token\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz\" (UID: \"75776f35-e39d-4823-ac2a-61e5b4e69619\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"
Oct 30 00:45:35.694053 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.694005    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75776f35-e39d-4823-ac2a-61e5b4e69619-istiod-ca-cert\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz\" (UID: \"75776f35-e39d-4823-ac2a-61e5b4e69619\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"
Oct 30 00:45:35.694158 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.694086    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75776f35-e39d-4823-ac2a-61e5b4e69619-credential-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz\" (UID: \"75776f35-e39d-4823-ac2a-61e5b4e69619\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"
Oct 30 00:45:35.694158 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.694114    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59s52\" (UniqueName: \"kubernetes.io/projected/75776f35-e39d-4823-ac2a-61e5b4e69619-kube-api-access-59s52\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz\" (UID: \"75776f35-e39d-4823-ac2a-61e5b4e69619\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"
Oct 30 00:45:35.694158 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.694147    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75776f35-e39d-4823-ac2a-61e5b4e69619-workload-certs\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz\" (UID: \"75776f35-e39d-4823-ac2a-61e5b4e69619\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"
Oct 30 00:45:35.694304 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.694159    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75776f35-e39d-4823-ac2a-61e5b4e69619-workload-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz\" (UID: \"75776f35-e39d-4823-ac2a-61e5b4e69619\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"
Oct 30 00:45:35.694538 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.694508    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75776f35-e39d-4823-ac2a-61e5b4e69619-workload-certs\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz\" (UID: \"75776f35-e39d-4823-ac2a-61e5b4e69619\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"
Oct 30 00:45:35.694740 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.694716    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75776f35-e39d-4823-ac2a-61e5b4e69619-istio-data\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz\" (UID: \"75776f35-e39d-4823-ac2a-61e5b4e69619\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"
Oct 30 00:45:35.694872 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.694719    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75776f35-e39d-4823-ac2a-61e5b4e69619-credential-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz\" (UID: \"75776f35-e39d-4823-ac2a-61e5b4e69619\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"
Oct 30 00:45:35.695147 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.695128    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75776f35-e39d-4823-ac2a-61e5b4e69619-istiod-ca-cert\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz\" (UID: \"75776f35-e39d-4823-ac2a-61e5b4e69619\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"
Oct 30 00:45:35.696938 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.696913    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75776f35-e39d-4823-ac2a-61e5b4e69619-istio-envoy\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz\" (UID: \"75776f35-e39d-4823-ac2a-61e5b4e69619\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"
Oct 30 00:45:35.697251 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.697229    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75776f35-e39d-4823-ac2a-61e5b4e69619-istio-podinfo\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz\" (UID: \"75776f35-e39d-4823-ac2a-61e5b4e69619\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"
Oct 30 00:45:35.717725 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.717699    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59s52\" (UniqueName: \"kubernetes.io/projected/75776f35-e39d-4823-ac2a-61e5b4e69619-kube-api-access-59s52\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz\" (UID: \"75776f35-e39d-4823-ac2a-61e5b4e69619\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"
Oct 30 00:45:35.717884 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.717707    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75776f35-e39d-4823-ac2a-61e5b4e69619-istio-token\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz\" (UID: \"75776f35-e39d-4823-ac2a-61e5b4e69619\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"
Oct 30 00:45:35.806273 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.806243    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"]
Oct 30 00:45:35.807834 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:45:35.807806    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8bdca6c_102d_4aaa_ab76_16a9123d6a74.slice/crio-838a2a7c6018f42eaf1582f095167dd033fbf9e609ad5a884389fbb473eb7e2c WatchSource:0}: Error finding container 838a2a7c6018f42eaf1582f095167dd033fbf9e609ad5a884389fbb473eb7e2c: Status 404 returned error can't find the container with id 838a2a7c6018f42eaf1582f095167dd033fbf9e609ad5a884389fbb473eb7e2c
Oct 30 00:45:35.810387 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.810350    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:45:35.810459 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.810422    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:45:35.810511 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.810457    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:45:35.829748 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:35.825845    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"
Oct 30 00:45:36.134931 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:36.134894    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"]
Oct 30 00:45:36.137655 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:45:36.137615    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75776f35_e39d_4823_ac2a_61e5b4e69619.slice/crio-b4f7a46546a1c174e4a1c619704db20e4d456e34692ec0ed73f31cf2d3fb829e WatchSource:0}: Error finding container b4f7a46546a1c174e4a1c619704db20e4d456e34692ec0ed73f31cf2d3fb829e: Status 404 returned error can't find the container with id b4f7a46546a1c174e4a1c619704db20e4d456e34692ec0ed73f31cf2d3fb829e
Oct 30 00:45:36.140689 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:36.140649    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:45:36.140794 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:36.140727    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:45:36.140794 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:36.140757    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:45:36.271437 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:36.271406    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz" event={"ID":"75776f35-e39d-4823-ac2a-61e5b4e69619","Type":"ContainerStarted","Data":"21274a5260f967854a3e02669bdef052286a44ce05fe8daa5da3648ff98e5841"}
Oct 30 00:45:36.271641 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:36.271468    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz" event={"ID":"75776f35-e39d-4823-ac2a-61e5b4e69619","Type":"ContainerStarted","Data":"b4f7a46546a1c174e4a1c619704db20e4d456e34692ec0ed73f31cf2d3fb829e"}
Oct 30 00:45:36.273375 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:36.273339    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9" event={"ID":"7afd2398-4294-4236-95d4-82b86a99ca43","Type":"ContainerStarted","Data":"e0e6169ef6d8353b19049d09b7d5b9e43eadd484e6010553454711b2f32ec8f5"}
Oct 30 00:45:36.273510 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:36.273386    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9" event={"ID":"7afd2398-4294-4236-95d4-82b86a99ca43","Type":"ContainerStarted","Data":"ad0f2229a433d598d4833444c97a0080708f723e3fb2587f1269af74d16eb580"}
Oct 30 00:45:36.275156 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:36.275127    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg" event={"ID":"a8bdca6c-102d-4aaa-ab76-16a9123d6a74","Type":"ContainerStarted","Data":"434b71c4baabd19f38964a7311756922c8f8934b3b6c8beebe7df7898d3d01d8"}
Oct 30 00:45:36.275269 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:36.275170    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg" event={"ID":"a8bdca6c-102d-4aaa-ab76-16a9123d6a74","Type":"ContainerStarted","Data":"838a2a7c6018f42eaf1582f095167dd033fbf9e609ad5a884389fbb473eb7e2c"}
Oct 30 00:45:36.315617 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:36.315567    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz" podStartSLOduration=1.315552808 podStartE2EDuration="1.315552808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:45:36.314736836 +0000 UTC m=+826.702413226" watchObservedRunningTime="2026-10-30 00:45:36.315552808 +0000 UTC m=+826.703229196"
Oct 30 00:45:36.359574 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:36.359510    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9" podStartSLOduration=2.359496665 podStartE2EDuration="2.359496665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:45:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:45:36.356935103 +0000 UTC m=+826.744611493" watchObservedRunningTime="2026-10-30 00:45:36.359496665 +0000 UTC m=+826.747173054"
Oct 30 00:45:36.401380 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:36.401252    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg" podStartSLOduration=1.401233169 podStartE2EDuration="1.401233169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:45:36.399548075 +0000 UTC m=+826.787224478" watchObservedRunningTime="2026-10-30 00:45:36.401233169 +0000 UTC m=+826.788909564"
Oct 30 00:45:36.484491 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:36.484460    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"
Oct 30 00:45:36.485758 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:36.485726    4393 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 00:45:36.486136 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:36.485801    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg" podUID="a8bdca6c-102d-4aaa-ab76-16a9123d6a74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 00:45:36.828024 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:36.827875    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"
Oct 30 00:45:37.253813 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:37.249202    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"
Oct 30 00:45:37.253813 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:37.250429    4393 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:45:37.253813 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:37.250478    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9" podUID="7afd2398-4294-4236-95d4-82b86a99ca43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 00:45:37.484622 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:37.484590    4393 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 00:45:37.484881 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:37.484852    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg" podUID="a8bdca6c-102d-4aaa-ab76-16a9123d6a74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 00:45:37.846806 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:37.841945    4393 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 00:45:37.846806 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:37.842015    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz" podUID="75776f35-e39d-4823-ac2a-61e5b4e69619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 00:45:38.250809 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:38.246873    4393 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:45:38.250809 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:38.246934    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9" podUID="7afd2398-4294-4236-95d4-82b86a99ca43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 00:45:38.484806 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:38.484717    4393 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 00:45:38.485319 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:38.484805    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg" podUID="a8bdca6c-102d-4aaa-ab76-16a9123d6a74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 00:45:38.858888 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:38.848155    4393 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 00:45:38.858888 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:38.848213    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz" podUID="75776f35-e39d-4823-ac2a-61e5b4e69619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 00:45:39.245805 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:39.240203    4393 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:45:39.245805 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:39.240278    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9" podUID="7afd2398-4294-4236-95d4-82b86a99ca43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 00:45:39.486813 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:39.486182    4393 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 00:45:39.486813 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:39.486259    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg" podUID="a8bdca6c-102d-4aaa-ab76-16a9123d6a74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 00:45:39.894823 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:39.876845    4393 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 00:45:39.912074 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:39.912005    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz" podUID="75776f35-e39d-4823-ac2a-61e5b4e69619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 00:45:40.238862 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:40.238133    4393 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:45:40.238862 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:40.238194    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9" podUID="7afd2398-4294-4236-95d4-82b86a99ca43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 00:45:40.494899 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:40.487374    4393 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 00:45:40.494899 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:40.487465    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg" podUID="a8bdca6c-102d-4aaa-ab76-16a9123d6a74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 00:45:40.829359 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:40.829215    4393 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 00:45:40.829638 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:40.829584    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz" podUID="75776f35-e39d-4823-ac2a-61e5b4e69619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 00:45:41.237194 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:41.237162    4393 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:45:41.237475 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:41.237448    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9" podUID="7afd2398-4294-4236-95d4-82b86a99ca43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 00:45:41.489868 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:41.485141    4393 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 00:45:41.489868 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:41.485205    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg" podUID="a8bdca6c-102d-4aaa-ab76-16a9123d6a74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 00:45:42.099503 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:42.099468    4393 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 00:45:42.118962 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:42.118919    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz" podUID="75776f35-e39d-4823-ac2a-61e5b4e69619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 00:45:42.247105 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:42.247066    4393 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:45:42.247423 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:42.247394    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9" podUID="7afd2398-4294-4236-95d4-82b86a99ca43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 00:45:42.484814 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:42.484363    4393 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 00:45:42.484814 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:42.484425    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg" podUID="a8bdca6c-102d-4aaa-ab76-16a9123d6a74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 00:45:42.926804 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:42.926415    4393 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 00:45:42.926804 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:42.926496    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz" podUID="75776f35-e39d-4823-ac2a-61e5b4e69619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 00:45:43.242000 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:43.241965    4393 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 00:45:43.242582 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:43.242550    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9" podUID="7afd2398-4294-4236-95d4-82b86a99ca43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 00:45:43.484837 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:43.484766    4393 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 00:45:43.485134 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:43.485104    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg" podUID="a8bdca6c-102d-4aaa-ab76-16a9123d6a74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 00:45:43.826669 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:43.826624    4393 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 00:45:43.826876 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:43.826695    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz" podUID="75776f35-e39d-4823-ac2a-61e5b4e69619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 00:45:44.393197 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:44.393164    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"
Oct 30 00:45:44.396272 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:44.396249    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"
Oct 30 00:45:44.398823 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:44.398804    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"
Oct 30 00:45:45.221650 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:45.221620    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"
Oct 30 00:45:45.222893 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:45.222873    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"
Oct 30 00:45:45.223902 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:45.223874    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"
Oct 30 00:45:45.289088 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:45.289056    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"
Oct 30 00:45:45.289352 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:45.289333    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"
Oct 30 00:45:45.290153 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:45.290137    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"
Oct 30 00:45:56.510920 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:45:56.510884    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:45:59.911603 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:59.911550    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"]
Oct 30 00:45:59.917320 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:59.917293    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"
Oct 30 00:45:59.955548 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:59.955519    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"]
Oct 30 00:45:59.967708 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:59.967670    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b4591e-8713-40c5-b33c-e6505356ee29-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kfpcv\" (UID: \"a5b4591e-8713-40c5-b33c-e6505356ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"
Oct 30 00:45:59.967906 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:59.967717    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b4591e-8713-40c5-b33c-e6505356ee29-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kfpcv\" (UID: \"a5b4591e-8713-40c5-b33c-e6505356ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"
Oct 30 00:45:59.967906 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:59.967739    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5b4591e-8713-40c5-b33c-e6505356ee29-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kfpcv\" (UID: \"a5b4591e-8713-40c5-b33c-e6505356ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"
Oct 30 00:45:59.968036 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:59.967901    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5b4591e-8713-40c5-b33c-e6505356ee29-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kfpcv\" (UID: \"a5b4591e-8713-40c5-b33c-e6505356ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"
Oct 30 00:45:59.968036 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:59.967932    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5b4591e-8713-40c5-b33c-e6505356ee29-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kfpcv\" (UID: \"a5b4591e-8713-40c5-b33c-e6505356ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"
Oct 30 00:45:59.968036 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:59.967955    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5b4591e-8713-40c5-b33c-e6505356ee29-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kfpcv\" (UID: \"a5b4591e-8713-40c5-b33c-e6505356ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"
Oct 30 00:45:59.968036 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:59.968001    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5b4591e-8713-40c5-b33c-e6505356ee29-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kfpcv\" (UID: \"a5b4591e-8713-40c5-b33c-e6505356ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"
Oct 30 00:45:59.968251 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:59.968076    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxfkf\" (UniqueName: \"kubernetes.io/projected/a5b4591e-8713-40c5-b33c-e6505356ee29-kube-api-access-mxfkf\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kfpcv\" (UID: \"a5b4591e-8713-40c5-b33c-e6505356ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"
Oct 30 00:45:59.968251 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:45:59.968139    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5b4591e-8713-40c5-b33c-e6505356ee29-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kfpcv\" (UID: \"a5b4591e-8713-40c5-b33c-e6505356ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"
Oct 30 00:46:00.068823 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:00.068758    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b4591e-8713-40c5-b33c-e6505356ee29-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kfpcv\" (UID: \"a5b4591e-8713-40c5-b33c-e6505356ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"
Oct 30 00:46:00.069002 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:00.068849    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b4591e-8713-40c5-b33c-e6505356ee29-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kfpcv\" (UID: \"a5b4591e-8713-40c5-b33c-e6505356ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"
Oct 30 00:46:00.069002 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:00.068876    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5b4591e-8713-40c5-b33c-e6505356ee29-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kfpcv\" (UID: \"a5b4591e-8713-40c5-b33c-e6505356ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"
Oct 30 00:46:00.069002 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:00.068953    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5b4591e-8713-40c5-b33c-e6505356ee29-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kfpcv\" (UID: \"a5b4591e-8713-40c5-b33c-e6505356ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"
Oct 30 00:46:00.069002 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:00.068977    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5b4591e-8713-40c5-b33c-e6505356ee29-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kfpcv\" (UID: \"a5b4591e-8713-40c5-b33c-e6505356ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"
Oct 30 00:46:00.069002 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:00.068996    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5b4591e-8713-40c5-b33c-e6505356ee29-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kfpcv\" (UID: \"a5b4591e-8713-40c5-b33c-e6505356ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"
Oct 30 00:46:00.069279 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:00.069062    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5b4591e-8713-40c5-b33c-e6505356ee29-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kfpcv\" (UID: \"a5b4591e-8713-40c5-b33c-e6505356ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"
Oct 30 00:46:00.069279 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:00.069125    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxfkf\" (UniqueName: \"kubernetes.io/projected/a5b4591e-8713-40c5-b33c-e6505356ee29-kube-api-access-mxfkf\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kfpcv\" (UID: \"a5b4591e-8713-40c5-b33c-e6505356ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"
Oct 30 00:46:00.069279 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:00.069163    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b4591e-8713-40c5-b33c-e6505356ee29-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kfpcv\" (UID: \"a5b4591e-8713-40c5-b33c-e6505356ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"
Oct 30 00:46:00.069279 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:00.069179    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5b4591e-8713-40c5-b33c-e6505356ee29-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kfpcv\" (UID: \"a5b4591e-8713-40c5-b33c-e6505356ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"
Oct 30 00:46:00.069279 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:00.069241    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5b4591e-8713-40c5-b33c-e6505356ee29-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kfpcv\" (UID: \"a5b4591e-8713-40c5-b33c-e6505356ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"
Oct 30 00:46:00.069279 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:00.069272    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b4591e-8713-40c5-b33c-e6505356ee29-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kfpcv\" (UID: \"a5b4591e-8713-40c5-b33c-e6505356ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"
Oct 30 00:46:00.069602 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:00.069473    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5b4591e-8713-40c5-b33c-e6505356ee29-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kfpcv\" (UID: \"a5b4591e-8713-40c5-b33c-e6505356ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"
Oct 30 00:46:00.069764 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:00.069741    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5b4591e-8713-40c5-b33c-e6505356ee29-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kfpcv\" (UID: \"a5b4591e-8713-40c5-b33c-e6505356ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"
Oct 30 00:46:00.072012 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:00.071979    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5b4591e-8713-40c5-b33c-e6505356ee29-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kfpcv\" (UID: \"a5b4591e-8713-40c5-b33c-e6505356ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"
Oct 30 00:46:00.072307 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:00.072284    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5b4591e-8713-40c5-b33c-e6505356ee29-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kfpcv\" (UID: \"a5b4591e-8713-40c5-b33c-e6505356ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"
Oct 30 00:46:00.091919 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:00.091893    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5b4591e-8713-40c5-b33c-e6505356ee29-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kfpcv\" (UID: \"a5b4591e-8713-40c5-b33c-e6505356ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"
Oct 30 00:46:00.092075 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:00.092055    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxfkf\" (UniqueName: \"kubernetes.io/projected/a5b4591e-8713-40c5-b33c-e6505356ee29-kube-api-access-mxfkf\") pod \"maas-default-gateway-openshift-default-7d9897b54c-kfpcv\" (UID: \"a5b4591e-8713-40c5-b33c-e6505356ee29\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"
Oct 30 00:46:00.260939 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:00.260106    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"
Oct 30 00:46:00.580950 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:00.580928    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"]
Oct 30 00:46:00.581617 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:46:00.581592    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5b4591e_8713_40c5_b33c_e6505356ee29.slice/crio-91125411b9d28215ed8c80f0a8b6518c6b2ba2f8609f21ff007b5731f1fe4ab3 WatchSource:0}: Error finding container 91125411b9d28215ed8c80f0a8b6518c6b2ba2f8609f21ff007b5731f1fe4ab3: Status 404 returned error can't find the container with id 91125411b9d28215ed8c80f0a8b6518c6b2ba2f8609f21ff007b5731f1fe4ab3
Oct 30 00:46:00.583950 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:00.583916    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:46:00.584036 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:00.583982    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:46:00.584036 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:00.584011    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:46:01.535695 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:01.535658    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv" event={"ID":"a5b4591e-8713-40c5-b33c-e6505356ee29","Type":"ContainerStarted","Data":"0463d565486a83116c8f7d135d7ac13b1d0cac60f488c2b116caeddda4a5959a"}
Oct 30 00:46:01.536252 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:01.535720    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv" event={"ID":"a5b4591e-8713-40c5-b33c-e6505356ee29","Type":"ContainerStarted","Data":"91125411b9d28215ed8c80f0a8b6518c6b2ba2f8609f21ff007b5731f1fe4ab3"}
Oct 30 00:46:01.594677 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:01.594619    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv" podStartSLOduration=2.594606019 podStartE2EDuration="2.594606019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:45:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:01.580269711 +0000 UTC m=+851.967946091" watchObservedRunningTime="2026-10-30 00:46:01.594606019 +0000 UTC m=+851.982282408"
Oct 30 00:46:02.261712 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:02.261679    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"
Oct 30 00:46:02.263579 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:02.263552    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kfpcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 00:46:02.263697 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:02.263610    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv" podUID="a5b4591e-8713-40c5-b33c-e6505356ee29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 00:46:03.261547 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:03.261516    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kfpcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 00:46:03.261977 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:03.261567    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv" podUID="a5b4591e-8713-40c5-b33c-e6505356ee29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 00:46:04.280805 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:04.280622    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kfpcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 00:46:04.280805 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:04.280725    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv" podUID="a5b4591e-8713-40c5-b33c-e6505356ee29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 00:46:05.261686 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:05.261647    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kfpcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 00:46:05.261899 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:05.261718    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv" podUID="a5b4591e-8713-40c5-b33c-e6505356ee29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 00:46:06.261460 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:06.261426    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kfpcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 00:46:06.261861 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:06.261489    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv" podUID="a5b4591e-8713-40c5-b33c-e6505356ee29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 00:46:07.261502 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:07.261469    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kfpcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 00:46:07.261997 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:07.261557    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv" podUID="a5b4591e-8713-40c5-b33c-e6505356ee29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 00:46:08.261466 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:08.261433    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kfpcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 00:46:08.261641 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:08.261495    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv" podUID="a5b4591e-8713-40c5-b33c-e6505356ee29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 00:46:09.521733 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:09.521671    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"
Oct 30 00:46:09.522325 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:09.522155    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"
Oct 30 00:46:09.523132 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:09.523113    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"
Oct 30 00:46:20.543957 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.543925    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"]
Oct 30 00:46:20.549658 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.549635    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"
Oct 30 00:46:20.582301 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.582269    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"]
Oct 30 00:46:20.627306 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.627266    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f42086fc-bfe1-4ded-be68-a54e07853fd2-istiod-ca-cert\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg\" (UID: \"f42086fc-bfe1-4ded-be68-a54e07853fd2\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"
Oct 30 00:46:20.627480 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.627323    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f42086fc-bfe1-4ded-be68-a54e07853fd2-workload-certs\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg\" (UID: \"f42086fc-bfe1-4ded-be68-a54e07853fd2\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"
Oct 30 00:46:20.627480 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.627404    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f42086fc-bfe1-4ded-be68-a54e07853fd2-istio-token\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg\" (UID: \"f42086fc-bfe1-4ded-be68-a54e07853fd2\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"
Oct 30 00:46:20.627575 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.627505    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m5bk\" (UniqueName: \"kubernetes.io/projected/f42086fc-bfe1-4ded-be68-a54e07853fd2-kube-api-access-9m5bk\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg\" (UID: \"f42086fc-bfe1-4ded-be68-a54e07853fd2\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"
Oct 30 00:46:20.627610 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.627598    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f42086fc-bfe1-4ded-be68-a54e07853fd2-istio-data\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg\" (UID: \"f42086fc-bfe1-4ded-be68-a54e07853fd2\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"
Oct 30 00:46:20.627671 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.627655    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f42086fc-bfe1-4ded-be68-a54e07853fd2-workload-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg\" (UID: \"f42086fc-bfe1-4ded-be68-a54e07853fd2\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"
Oct 30 00:46:20.627735 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.627720    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f42086fc-bfe1-4ded-be68-a54e07853fd2-istio-envoy\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg\" (UID: \"f42086fc-bfe1-4ded-be68-a54e07853fd2\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"
Oct 30 00:46:20.627836 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.627817    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f42086fc-bfe1-4ded-be68-a54e07853fd2-credential-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg\" (UID: \"f42086fc-bfe1-4ded-be68-a54e07853fd2\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"
Oct 30 00:46:20.627933 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.627918    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f42086fc-bfe1-4ded-be68-a54e07853fd2-istio-podinfo\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg\" (UID: \"f42086fc-bfe1-4ded-be68-a54e07853fd2\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"
Oct 30 00:46:20.681046 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.681014    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"]
Oct 30 00:46:20.685143 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.685127    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"
Oct 30 00:46:20.710591 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.710560    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"]
Oct 30 00:46:20.728872 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.728845    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m5bk\" (UniqueName: \"kubernetes.io/projected/f42086fc-bfe1-4ded-be68-a54e07853fd2-kube-api-access-9m5bk\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg\" (UID: \"f42086fc-bfe1-4ded-be68-a54e07853fd2\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"
Oct 30 00:46:20.729093 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.729034    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f42086fc-bfe1-4ded-be68-a54e07853fd2-istio-data\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg\" (UID: \"f42086fc-bfe1-4ded-be68-a54e07853fd2\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"
Oct 30 00:46:20.729190 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.729127    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-workload-certs\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd\" (UID: \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"
Oct 30 00:46:20.729190 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.729180    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-istio-data\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd\" (UID: \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"
Oct 30 00:46:20.729345 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.729289    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f42086fc-bfe1-4ded-be68-a54e07853fd2-workload-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg\" (UID: \"f42086fc-bfe1-4ded-be68-a54e07853fd2\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"
Oct 30 00:46:20.729345 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.729328    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-workload-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd\" (UID: \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"
Oct 30 00:46:20.729493 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.729392    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-istio-envoy\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd\" (UID: \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"
Oct 30 00:46:20.729493 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.729429    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f42086fc-bfe1-4ded-be68-a54e07853fd2-istio-envoy\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg\" (UID: \"f42086fc-bfe1-4ded-be68-a54e07853fd2\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"
Oct 30 00:46:20.729493 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.729458    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f42086fc-bfe1-4ded-be68-a54e07853fd2-credential-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg\" (UID: \"f42086fc-bfe1-4ded-be68-a54e07853fd2\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"
Oct 30 00:46:20.729652 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.729492    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f42086fc-bfe1-4ded-be68-a54e07853fd2-istio-data\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg\" (UID: \"f42086fc-bfe1-4ded-be68-a54e07853fd2\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"
Oct 30 00:46:20.729652 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.729558    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-istio-token\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd\" (UID: \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"
Oct 30 00:46:20.729652 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.729620    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f42086fc-bfe1-4ded-be68-a54e07853fd2-istio-podinfo\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg\" (UID: \"f42086fc-bfe1-4ded-be68-a54e07853fd2\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"
Oct 30 00:46:20.729839 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.729665    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f42086fc-bfe1-4ded-be68-a54e07853fd2-istiod-ca-cert\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg\" (UID: \"f42086fc-bfe1-4ded-be68-a54e07853fd2\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"
Oct 30 00:46:20.729839 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.729695    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-istio-podinfo\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd\" (UID: \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"
Oct 30 00:46:20.730286 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.730249    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f42086fc-bfe1-4ded-be68-a54e07853fd2-workload-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg\" (UID: \"f42086fc-bfe1-4ded-be68-a54e07853fd2\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"
Oct 30 00:46:20.730385 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.730316    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f42086fc-bfe1-4ded-be68-a54e07853fd2-istiod-ca-cert\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg\" (UID: \"f42086fc-bfe1-4ded-be68-a54e07853fd2\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"
Oct 30 00:46:20.730385 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.730333    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4phr4\" (UniqueName: \"kubernetes.io/projected/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-kube-api-access-4phr4\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd\" (UID: \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"
Oct 30 00:46:20.731814 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.730558    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f42086fc-bfe1-4ded-be68-a54e07853fd2-workload-certs\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg\" (UID: \"f42086fc-bfe1-4ded-be68-a54e07853fd2\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"
Oct 30 00:46:20.731950 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.731865    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f42086fc-bfe1-4ded-be68-a54e07853fd2-istio-token\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg\" (UID: \"f42086fc-bfe1-4ded-be68-a54e07853fd2\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"
Oct 30 00:46:20.731950 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.731899    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-credential-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd\" (UID: \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"
Oct 30 00:46:20.732055 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.731950    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-istiod-ca-cert\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd\" (UID: \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"
Oct 30 00:46:20.732164 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.731672    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f42086fc-bfe1-4ded-be68-a54e07853fd2-workload-certs\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg\" (UID: \"f42086fc-bfe1-4ded-be68-a54e07853fd2\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"
Oct 30 00:46:20.732164 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.730940    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f42086fc-bfe1-4ded-be68-a54e07853fd2-credential-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg\" (UID: \"f42086fc-bfe1-4ded-be68-a54e07853fd2\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"
Oct 30 00:46:20.733329 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.733304    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f42086fc-bfe1-4ded-be68-a54e07853fd2-istio-envoy\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg\" (UID: \"f42086fc-bfe1-4ded-be68-a54e07853fd2\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"
Oct 30 00:46:20.733534 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.733517    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f42086fc-bfe1-4ded-be68-a54e07853fd2-istio-podinfo\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg\" (UID: \"f42086fc-bfe1-4ded-be68-a54e07853fd2\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"
Oct 30 00:46:20.751866 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.751841    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m5bk\" (UniqueName: \"kubernetes.io/projected/f42086fc-bfe1-4ded-be68-a54e07853fd2-kube-api-access-9m5bk\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg\" (UID: \"f42086fc-bfe1-4ded-be68-a54e07853fd2\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"
Oct 30 00:46:20.754792 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.754745    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f42086fc-bfe1-4ded-be68-a54e07853fd2-istio-token\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg\" (UID: \"f42086fc-bfe1-4ded-be68-a54e07853fd2\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"
Oct 30 00:46:20.833668 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.833586    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-istio-podinfo\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd\" (UID: \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"
Oct 30 00:46:20.833668 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.833622    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4phr4\" (UniqueName: \"kubernetes.io/projected/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-kube-api-access-4phr4\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd\" (UID: \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"
Oct 30 00:46:20.833668 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.833647    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-credential-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd\" (UID: \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"
Oct 30 00:46:20.833668 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.833670    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-istiod-ca-cert\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd\" (UID: \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"
Oct 30 00:46:20.834003 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.833742    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-workload-certs\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd\" (UID: \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"
Oct 30 00:46:20.834003 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.833765    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-istio-data\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd\" (UID: \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"
Oct 30 00:46:20.834003 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.833854    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-workload-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd\" (UID: \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"
Oct 30 00:46:20.834003 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.833895    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-istio-envoy\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd\" (UID: \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"
Oct 30 00:46:20.834003 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.833996    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-istio-token\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd\" (UID: \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"
Oct 30 00:46:20.834259 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.834145    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-credential-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd\" (UID: \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"
Oct 30 00:46:20.834332 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.834302    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-workload-certs\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd\" (UID: \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"
Oct 30 00:46:20.834529 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.834472    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-istio-data\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd\" (UID: \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"
Oct 30 00:46:20.834529 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.834521    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-workload-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd\" (UID: \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"
Oct 30 00:46:20.834757 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.834740    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-istiod-ca-cert\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd\" (UID: \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"
Oct 30 00:46:20.836237 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.836213    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-istio-envoy\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd\" (UID: \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"
Oct 30 00:46:20.836346 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.836329    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-istio-podinfo\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd\" (UID: \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"
Oct 30 00:46:20.856227 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.856200    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-istio-token\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd\" (UID: \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"
Oct 30 00:46:20.856326 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.856242    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4phr4\" (UniqueName: \"kubernetes.io/projected/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-kube-api-access-4phr4\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd\" (UID: \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"
Oct 30 00:46:20.864217 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:20.864199    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"
Oct 30 00:46:21.000597 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:21.000562    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"
Oct 30 00:46:21.197619 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:21.197569    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"]
Oct 30 00:46:21.220070 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:21.220026    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:46:21.220216 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:21.220146    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:46:21.220216 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:21.220186    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:46:21.307903 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:21.307872    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"]
Oct 30 00:46:21.308456 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:46:21.308423    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod394e8f0b_d96a_44ec_9cdf_a1266ee49a91.slice/crio-37cc592955e5ec8f31d695af509db5fdec5cf0082828ce3e92e349866987d14e WatchSource:0}: Error finding container 37cc592955e5ec8f31d695af509db5fdec5cf0082828ce3e92e349866987d14e: Status 404 returned error can't find the container with id 37cc592955e5ec8f31d695af509db5fdec5cf0082828ce3e92e349866987d14e
Oct 30 00:46:21.311171 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:21.311140    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:46:21.311261 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:21.311218    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:46:21.311329 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:21.311260    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:46:21.737365 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:21.737313    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd" event={"ID":"394e8f0b-d96a-44ec-9cdf-a1266ee49a91","Type":"ContainerStarted","Data":"2dd986d6f0568810d411879ce7fcbf8688aab0d8704561e359b76b558a298306"}
Oct 30 00:46:21.738035 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:21.738001    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd" event={"ID":"394e8f0b-d96a-44ec-9cdf-a1266ee49a91","Type":"ContainerStarted","Data":"37cc592955e5ec8f31d695af509db5fdec5cf0082828ce3e92e349866987d14e"}
Oct 30 00:46:21.745902 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:21.745862    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg" event={"ID":"f42086fc-bfe1-4ded-be68-a54e07853fd2","Type":"ContainerStarted","Data":"629f5bd40027d52b5fe1e2de0fba03d2ee3bc356f32e72298cd3221d045be7dc"}
Oct 30 00:46:21.746088 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:21.745922    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg" event={"ID":"f42086fc-bfe1-4ded-be68-a54e07853fd2","Type":"ContainerStarted","Data":"f7e50d68227dc30f74e1fec9c313bdac77199f415b6c25794858898bdd7eb89c"}
Oct 30 00:46:21.783216 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:21.783165    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd" podStartSLOduration=1.78315095 podStartE2EDuration="1.78315095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:21.782860196 +0000 UTC m=+872.170536584" watchObservedRunningTime="2026-10-30 00:46:21.78315095 +0000 UTC m=+872.170827339"
Oct 30 00:46:21.837507 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:21.837451    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg" podStartSLOduration=1.837432376 podStartE2EDuration="1.837432376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:21.836144696 +0000 UTC m=+872.223821099" watchObservedRunningTime="2026-10-30 00:46:21.837432376 +0000 UTC m=+872.225108765"
Oct 30 00:46:21.864359 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:21.864314    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"
Oct 30 00:46:22.008865 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:22.002936    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"
Oct 30 00:46:22.012808 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:22.010917    4393 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 00:46:22.012808 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:22.010982    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd" podUID="394e8f0b-d96a-44ec-9cdf-a1266ee49a91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 00:46:22.866339 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:22.866303    4393 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 00:46:22.866810 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:22.866357    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg" podUID="f42086fc-bfe1-4ded-be68-a54e07853fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 00:46:23.005817 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:23.001906    4393 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 00:46:23.005817 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:23.001978    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd" podUID="394e8f0b-d96a-44ec-9cdf-a1266ee49a91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 00:46:23.865136 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:23.865104    4393 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 00:46:23.865377 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:23.865356    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg" podUID="f42086fc-bfe1-4ded-be68-a54e07853fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 00:46:24.001894 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:24.001858    4393 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 00:46:24.002498 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:24.002456    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd" podUID="394e8f0b-d96a-44ec-9cdf-a1266ee49a91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 00:46:24.865343 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:24.865309    4393 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 00:46:24.865532 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:24.865392    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg" podUID="f42086fc-bfe1-4ded-be68-a54e07853fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 00:46:25.001484 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:25.001453    4393 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 00:46:25.001657 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:25.001505    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd" podUID="394e8f0b-d96a-44ec-9cdf-a1266ee49a91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 00:46:25.867992 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:25.867958    4393 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 00:46:25.868808 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:25.868740    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg" podUID="f42086fc-bfe1-4ded-be68-a54e07853fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 00:46:26.026804 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:26.001219    4393 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 00:46:26.026804 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:26.001277    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd" podUID="394e8f0b-d96a-44ec-9cdf-a1266ee49a91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 00:46:26.058992 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:26.057163    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"]
Oct 30 00:46:26.062233 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:26.062149    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"]
Oct 30 00:46:26.062233 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:26.062234    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"
Oct 30 00:46:26.078842 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:26.078012    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-52650f\"/\"kube-root-ca.crt\""
Oct 30 00:46:26.087966 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:26.087946    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-52650f\"/\"default-dockercfg-7qbxd\""
Oct 30 00:46:26.087966 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:26.087950    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-52650f\"/\"openshift-service-ca.crt\""
Oct 30 00:46:26.088173 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:26.088069    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-52650f\"/\"distinct-2-w0-52650f-kserve-self-signed-certs\""
Oct 30 00:46:26.151901 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:26.148227    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-home\") pod \"distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g\" (UID: \"fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d\") " pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"
Oct 30 00:46:26.151901 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:26.148554    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-dshm\") pod \"distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g\" (UID: \"fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d\") " pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"
Oct 30 00:46:26.151901 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:26.148589    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67zpt\" (UniqueName: \"kubernetes.io/projected/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-kube-api-access-67zpt\") pod \"distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g\" (UID: \"fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d\") " pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"
Oct 30 00:46:26.151901 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:26.148637    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-tls-certs\") pod \"distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g\" (UID: \"fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d\") " pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"
Oct 30 00:46:26.151901 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:26.148698    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-model-cache\") pod \"distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g\" (UID: \"fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d\") " pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"
Oct 30 00:46:26.151901 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:26.148805    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-tmp-dir\") pod \"distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g\" (UID: \"fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d\") " pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"
Oct 30 00:46:26.263872 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:26.262570    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-dshm\") pod \"distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g\" (UID: \"fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d\") " pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"
Oct 30 00:46:26.263872 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:26.262620    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67zpt\" (UniqueName: \"kubernetes.io/projected/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-kube-api-access-67zpt\") pod \"distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g\" (UID: \"fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d\") " pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"
Oct 30 00:46:26.263872 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:26.262680    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-tls-certs\") pod \"distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g\" (UID: \"fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d\") " pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"
Oct 30 00:46:26.263872 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:26.262747    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-model-cache\") pod \"distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g\" (UID: \"fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d\") " pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"
Oct 30 00:46:26.263872 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:26.262862    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-tmp-dir\") pod \"distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g\" (UID: \"fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d\") " pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"
Oct 30 00:46:26.263872 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:26.262909    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-home\") pod \"distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g\" (UID: \"fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d\") " pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"
Oct 30 00:46:26.263872 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:26.263421    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-home\") pod \"distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g\" (UID: \"fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d\") " pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"
Oct 30 00:46:26.266322 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:26.266016    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-model-cache\") pod \"distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g\" (UID: \"fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d\") " pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"
Oct 30 00:46:26.266322 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:26.266271    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-tmp-dir\") pod \"distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g\" (UID: \"fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d\") " pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"
Oct 30 00:46:26.288904 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:26.280586    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-tls-certs\") pod \"distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g\" (UID: \"fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d\") " pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"
Oct 30 00:46:26.302137 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:26.302074    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67zpt\" (UniqueName: \"kubernetes.io/projected/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-kube-api-access-67zpt\") pod \"distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g\" (UID: \"fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d\") " pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"
Oct 30 00:46:26.322049 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:26.321099    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-dshm\") pod \"distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g\" (UID: \"fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d\") " pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"
Oct 30 00:46:26.386298 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:26.386013    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"
Oct 30 00:46:26.569308 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:46:26.568180    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:46:26.810734 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:26.810624    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"]
Oct 30 00:46:26.852756 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:46:26.851085    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc843ff2_b1a1_4e3c_a7ee_2b349a074b4d.slice/crio-6bfb6799e0c65bfd1f02da4231da4507ba02e5c8706e0ace81259a09e48be72d WatchSource:0}: Error finding container 6bfb6799e0c65bfd1f02da4231da4507ba02e5c8706e0ace81259a09e48be72d: Status 404 returned error can't find the container with id 6bfb6799e0c65bfd1f02da4231da4507ba02e5c8706e0ace81259a09e48be72d
Oct 30 00:46:26.866722 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:26.865332    4393 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 00:46:26.866722 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:26.865390    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg" podUID="f42086fc-bfe1-4ded-be68-a54e07853fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 00:46:26.883639 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:26.883609    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g" event={"ID":"fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d","Type":"ContainerStarted","Data":"6bfb6799e0c65bfd1f02da4231da4507ba02e5c8706e0ace81259a09e48be72d"}
Oct 30 00:46:27.000995 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:27.000942    4393 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 00:46:27.001175 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:27.001008    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd" podUID="394e8f0b-d96a-44ec-9cdf-a1266ee49a91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 00:46:27.865613 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:27.865571    4393 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 00:46:27.865856 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:27.865662    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg" podUID="f42086fc-bfe1-4ded-be68-a54e07853fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 00:46:27.899639 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:27.899594    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g" event={"ID":"fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d","Type":"ContainerStarted","Data":"33dec75db61ea84de9262e5b1383f14e3c0c5b7df0fbadc5c454a6ead4476c9c"}
Oct 30 00:46:27.943504 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:27.943436    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g" podStartSLOduration=2.943415972 podStartE2EDuration="2.943415972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:27.938587444 +0000 UTC m=+878.326263833" watchObservedRunningTime="2026-10-30 00:46:27.943415972 +0000 UTC m=+878.331092361"
Oct 30 00:46:28.001182 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:28.001137    4393 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 00:46:28.001371 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:28.001220    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd" podUID="394e8f0b-d96a-44ec-9cdf-a1266ee49a91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 00:46:28.387330 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:28.387289    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"
Oct 30 00:46:28.869215 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:28.869188    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"
Oct 30 00:46:28.869656 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:28.869620    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"
Oct 30 00:46:28.870492 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:28.870475    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"
Oct 30 00:46:29.016318 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:29.016287    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"
Oct 30 00:46:29.016935 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:29.016910    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"
Oct 30 00:46:29.018577 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:29.018557    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"
Oct 30 00:46:35.481659 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:35.477933    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"]
Oct 30 00:46:35.489014 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:35.488980    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"]
Oct 30 00:46:35.489184 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:35.489099    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"
Oct 30 00:46:35.589824 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:35.589391    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c026d820-c4ea-4d97-97b7-18757b273b8e-workload-certs\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5\" (UID: \"c026d820-c4ea-4d97-97b7-18757b273b8e\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"
Oct 30 00:46:35.589824 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:35.589465    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gntrr\" (UniqueName: \"kubernetes.io/projected/c026d820-c4ea-4d97-97b7-18757b273b8e-kube-api-access-gntrr\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5\" (UID: \"c026d820-c4ea-4d97-97b7-18757b273b8e\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"
Oct 30 00:46:35.589824 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:35.589517    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c026d820-c4ea-4d97-97b7-18757b273b8e-workload-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5\" (UID: \"c026d820-c4ea-4d97-97b7-18757b273b8e\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"
Oct 30 00:46:35.589824 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:35.589559    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c026d820-c4ea-4d97-97b7-18757b273b8e-istio-podinfo\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5\" (UID: \"c026d820-c4ea-4d97-97b7-18757b273b8e\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"
Oct 30 00:46:35.589824 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:35.589677    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c026d820-c4ea-4d97-97b7-18757b273b8e-istio-envoy\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5\" (UID: \"c026d820-c4ea-4d97-97b7-18757b273b8e\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"
Oct 30 00:46:35.589824 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:35.589707    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c026d820-c4ea-4d97-97b7-18757b273b8e-istiod-ca-cert\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5\" (UID: \"c026d820-c4ea-4d97-97b7-18757b273b8e\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"
Oct 30 00:46:35.589824 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:35.589747    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c026d820-c4ea-4d97-97b7-18757b273b8e-istio-data\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5\" (UID: \"c026d820-c4ea-4d97-97b7-18757b273b8e\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"
Oct 30 00:46:35.593294 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:35.590486    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c026d820-c4ea-4d97-97b7-18757b273b8e-istio-token\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5\" (UID: \"c026d820-c4ea-4d97-97b7-18757b273b8e\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"
Oct 30 00:46:35.593294 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:35.590606    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c026d820-c4ea-4d97-97b7-18757b273b8e-credential-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5\" (UID: \"c026d820-c4ea-4d97-97b7-18757b273b8e\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"
Oct 30 00:46:35.692163 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:35.692123    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c026d820-c4ea-4d97-97b7-18757b273b8e-istio-envoy\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5\" (UID: \"c026d820-c4ea-4d97-97b7-18757b273b8e\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"
Oct 30 00:46:35.692163 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:35.692174    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c026d820-c4ea-4d97-97b7-18757b273b8e-istiod-ca-cert\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5\" (UID: \"c026d820-c4ea-4d97-97b7-18757b273b8e\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"
Oct 30 00:46:35.692441 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:35.692206    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c026d820-c4ea-4d97-97b7-18757b273b8e-istio-data\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5\" (UID: \"c026d820-c4ea-4d97-97b7-18757b273b8e\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"
Oct 30 00:46:35.692441 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:35.692260    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c026d820-c4ea-4d97-97b7-18757b273b8e-istio-token\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5\" (UID: \"c026d820-c4ea-4d97-97b7-18757b273b8e\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"
Oct 30 00:46:35.692441 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:35.692340    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c026d820-c4ea-4d97-97b7-18757b273b8e-credential-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5\" (UID: \"c026d820-c4ea-4d97-97b7-18757b273b8e\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"
Oct 30 00:46:35.692619 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:35.692458    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c026d820-c4ea-4d97-97b7-18757b273b8e-workload-certs\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5\" (UID: \"c026d820-c4ea-4d97-97b7-18757b273b8e\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"
Oct 30 00:46:35.692619 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:35.692484    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gntrr\" (UniqueName: \"kubernetes.io/projected/c026d820-c4ea-4d97-97b7-18757b273b8e-kube-api-access-gntrr\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5\" (UID: \"c026d820-c4ea-4d97-97b7-18757b273b8e\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"
Oct 30 00:46:35.692619 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:35.692526    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c026d820-c4ea-4d97-97b7-18757b273b8e-workload-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5\" (UID: \"c026d820-c4ea-4d97-97b7-18757b273b8e\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"
Oct 30 00:46:35.692619 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:35.692557    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c026d820-c4ea-4d97-97b7-18757b273b8e-istio-podinfo\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5\" (UID: \"c026d820-c4ea-4d97-97b7-18757b273b8e\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"
Oct 30 00:46:35.692889 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:35.692631    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c026d820-c4ea-4d97-97b7-18757b273b8e-istio-data\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5\" (UID: \"c026d820-c4ea-4d97-97b7-18757b273b8e\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"
Oct 30 00:46:35.692889 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:35.692855    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c026d820-c4ea-4d97-97b7-18757b273b8e-workload-certs\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5\" (UID: \"c026d820-c4ea-4d97-97b7-18757b273b8e\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"
Oct 30 00:46:35.693154 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:35.693127    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c026d820-c4ea-4d97-97b7-18757b273b8e-istiod-ca-cert\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5\" (UID: \"c026d820-c4ea-4d97-97b7-18757b273b8e\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"
Oct 30 00:46:35.693264 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:35.693241    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c026d820-c4ea-4d97-97b7-18757b273b8e-credential-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5\" (UID: \"c026d820-c4ea-4d97-97b7-18757b273b8e\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"
Oct 30 00:46:35.693517 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:35.693495    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c026d820-c4ea-4d97-97b7-18757b273b8e-workload-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5\" (UID: \"c026d820-c4ea-4d97-97b7-18757b273b8e\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"
Oct 30 00:46:35.695711 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:35.695687    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c026d820-c4ea-4d97-97b7-18757b273b8e-istio-envoy\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5\" (UID: \"c026d820-c4ea-4d97-97b7-18757b273b8e\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"
Oct 30 00:46:35.696070 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:35.696046    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c026d820-c4ea-4d97-97b7-18757b273b8e-istio-podinfo\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5\" (UID: \"c026d820-c4ea-4d97-97b7-18757b273b8e\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"
Oct 30 00:46:35.781409 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:35.780362    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c026d820-c4ea-4d97-97b7-18757b273b8e-istio-token\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5\" (UID: \"c026d820-c4ea-4d97-97b7-18757b273b8e\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"
Oct 30 00:46:35.802128 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:35.792176    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gntrr\" (UniqueName: \"kubernetes.io/projected/c026d820-c4ea-4d97-97b7-18757b273b8e-kube-api-access-gntrr\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5\" (UID: \"c026d820-c4ea-4d97-97b7-18757b273b8e\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"
Oct 30 00:46:35.816918 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:35.816880    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"
Oct 30 00:46:36.177852 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:36.177196    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"]
Oct 30 00:46:36.186816 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:36.186761    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:46:36.186937 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:36.186862    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:46:36.186937 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:36.186893    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:46:36.389129 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:36.387062    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"
Oct 30 00:46:36.421171 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:36.420822    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"
Oct 30 00:46:37.042748 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:37.042707    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5" event={"ID":"c026d820-c4ea-4d97-97b7-18757b273b8e","Type":"ContainerStarted","Data":"77d510a73c92dcdbf6c0a234a573c3a5e21dc5b1ad0925f347dfe43e5c70c051"}
Oct 30 00:46:37.043294 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:37.042816    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5" event={"ID":"c026d820-c4ea-4d97-97b7-18757b273b8e","Type":"ContainerStarted","Data":"74662283cec3a65f3b1e92a89c013b6dfb0d45f53315cde49e2eca229d081c90"}
Oct 30 00:46:37.060507 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:37.060482    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"
Oct 30 00:46:37.089481 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:37.089434    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5" podStartSLOduration=2.089420788 podStartE2EDuration="2.089420788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:37.086382383 +0000 UTC m=+887.474058772" watchObservedRunningTime="2026-10-30 00:46:37.089420788 +0000 UTC m=+887.477097177"
Oct 30 00:46:37.822154 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:37.822121    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"
Oct 30 00:46:37.828250 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:37.828221    4393 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 00:46:37.828472 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:37.828274    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5" podUID="c026d820-c4ea-4d97-97b7-18757b273b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 00:46:38.820340 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:38.820306    4393 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 00:46:38.820953 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:38.820922    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5" podUID="c026d820-c4ea-4d97-97b7-18757b273b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 00:46:39.823226 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:39.823184    4393 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 00:46:39.823959 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:39.823928    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5" podUID="c026d820-c4ea-4d97-97b7-18757b273b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 00:46:40.818226 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:40.818188    4393 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 00:46:40.818523 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:40.818496    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5" podUID="c026d820-c4ea-4d97-97b7-18757b273b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 00:46:41.821208 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:41.821167    4393 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 00:46:41.821966 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:41.821934    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5" podUID="c026d820-c4ea-4d97-97b7-18757b273b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 00:46:42.818410 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:42.818377    4393 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 00:46:42.818573 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:42.818434    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5" podUID="c026d820-c4ea-4d97-97b7-18757b273b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 00:46:43.881077 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:43.881046    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"
Oct 30 00:46:43.881594 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:43.881503    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"
Oct 30 00:46:43.882707 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:43.882684    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"
Oct 30 00:46:56.377250 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:56.377214    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"]
Oct 30 00:46:56.478804 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:56.472606    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"]
Oct 30 00:46:56.479147 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:56.479123    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"
Oct 30 00:46:56.573601 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:56.573551    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-tls-certs\") pod \"distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x\" (UID: \"28f3e4ff-5875-4e5d-8cab-60d1a5ba97be\") " pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"
Oct 30 00:46:56.573946 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:56.573901    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-home\") pod \"distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x\" (UID: \"28f3e4ff-5875-4e5d-8cab-60d1a5ba97be\") " pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"
Oct 30 00:46:56.574182 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:56.574165    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-dshm\") pod \"distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x\" (UID: \"28f3e4ff-5875-4e5d-8cab-60d1a5ba97be\") " pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"
Oct 30 00:46:56.581825 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:56.575429    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-021d6d\"/\"kube-root-ca.crt\""
Oct 30 00:46:56.581825 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:56.575527    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-021d6d\"/\"openshift-service-ca.crt\""
Oct 30 00:46:56.582185 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:56.582152    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-tmp-dir\") pod \"distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x\" (UID: \"28f3e4ff-5875-4e5d-8cab-60d1a5ba97be\") " pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"
Oct 30 00:46:56.582548 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:56.582509    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98tdz\" (UniqueName: \"kubernetes.io/projected/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-kube-api-access-98tdz\") pod \"distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x\" (UID: \"28f3e4ff-5875-4e5d-8cab-60d1a5ba97be\") " pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"
Oct 30 00:46:56.583027 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:56.582993    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-model-cache\") pod \"distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x\" (UID: \"28f3e4ff-5875-4e5d-8cab-60d1a5ba97be\") " pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"
Oct 30 00:46:56.591940 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:56.591909    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-021d6d\"/\"distinct-2-w6-021d6d-kserve-self-signed-certs\""
Oct 30 00:46:56.592270 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:56.592250    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-021d6d\"/\"default-dockercfg-mtkh7\""
Oct 30 00:46:56.727877 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:56.696547    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-tmp-dir\") pod \"distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x\" (UID: \"28f3e4ff-5875-4e5d-8cab-60d1a5ba97be\") " pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"
Oct 30 00:46:56.727877 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:56.724105    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98tdz\" (UniqueName: \"kubernetes.io/projected/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-kube-api-access-98tdz\") pod \"distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x\" (UID: \"28f3e4ff-5875-4e5d-8cab-60d1a5ba97be\") " pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"
Oct 30 00:46:56.727877 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:56.724212    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-model-cache\") pod \"distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x\" (UID: \"28f3e4ff-5875-4e5d-8cab-60d1a5ba97be\") " pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"
Oct 30 00:46:56.727877 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:56.724383    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-tls-certs\") pod \"distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x\" (UID: \"28f3e4ff-5875-4e5d-8cab-60d1a5ba97be\") " pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"
Oct 30 00:46:56.727877 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:56.724413    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-home\") pod \"distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x\" (UID: \"28f3e4ff-5875-4e5d-8cab-60d1a5ba97be\") " pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"
Oct 30 00:46:56.727877 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:56.724502    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-dshm\") pod \"distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x\" (UID: \"28f3e4ff-5875-4e5d-8cab-60d1a5ba97be\") " pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"
Oct 30 00:46:56.732803 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:56.732129    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-model-cache\") pod \"distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x\" (UID: \"28f3e4ff-5875-4e5d-8cab-60d1a5ba97be\") " pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"
Oct 30 00:46:56.732803 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:56.703366    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-tmp-dir\") pod \"distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x\" (UID: \"28f3e4ff-5875-4e5d-8cab-60d1a5ba97be\") " pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"
Oct 30 00:46:56.748111 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:56.747954    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-home\") pod \"distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x\" (UID: \"28f3e4ff-5875-4e5d-8cab-60d1a5ba97be\") " pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"
Oct 30 00:46:56.752439 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:56.752367    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-tls-certs\") pod \"distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x\" (UID: \"28f3e4ff-5875-4e5d-8cab-60d1a5ba97be\") " pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"
Oct 30 00:46:56.833804 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:56.833298    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-dshm\") pod \"distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x\" (UID: \"28f3e4ff-5875-4e5d-8cab-60d1a5ba97be\") " pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"
Oct 30 00:46:56.851470 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:56.851399    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98tdz\" (UniqueName: \"kubernetes.io/projected/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-kube-api-access-98tdz\") pod \"distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x\" (UID: \"28f3e4ff-5875-4e5d-8cab-60d1a5ba97be\") " pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"
Oct 30 00:46:56.965805 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:46:56.960715    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:46:57.110011 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:57.109924    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"
Oct 30 00:46:57.422742 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:57.422709    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"]
Oct 30 00:46:57.423342 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:46:57.423313    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28f3e4ff_5875_4e5d_8cab_60d1a5ba97be.slice/crio-8f8404868ee9a1191e4cf1958c83c84148721ad6a744ebf81d18dc36982acd91 WatchSource:0}: Error finding container 8f8404868ee9a1191e4cf1958c83c84148721ad6a744ebf81d18dc36982acd91: Status 404 returned error can't find the container with id 8f8404868ee9a1191e4cf1958c83c84148721ad6a744ebf81d18dc36982acd91
Oct 30 00:46:58.279454 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:58.279416    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x" event={"ID":"28f3e4ff-5875-4e5d-8cab-60d1a5ba97be","Type":"ContainerStarted","Data":"2f989378070201471da6c09ec75f086bafd5930ca1f2003b959fc4103c2a7c3b"}
Oct 30 00:46:58.279454 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:58.279464    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x" event={"ID":"28f3e4ff-5875-4e5d-8cab-60d1a5ba97be","Type":"ContainerStarted","Data":"8f8404868ee9a1191e4cf1958c83c84148721ad6a744ebf81d18dc36982acd91"}
Oct 30 00:46:58.319216 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:58.319164    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x" podStartSLOduration=2.319150786 podStartE2EDuration="2.319150786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:46:58.318646206 +0000 UTC m=+908.706322576" watchObservedRunningTime="2026-10-30 00:46:58.319150786 +0000 UTC m=+908.706827175"
Oct 30 00:46:59.110125 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:46:59.110093    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"
Oct 30 00:47:07.110834 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:07.110796    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"
Oct 30 00:47:07.124119 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:07.124083    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"
Oct 30 00:47:07.421411 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:07.421383    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"
Oct 30 00:47:20.485081 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.485046    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"]
Oct 30 00:47:20.489714 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.489691    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"
Oct 30 00:47:20.526902 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.526868    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"]
Oct 30 00:47:20.617297 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.617258    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26d33aa-399c-46f8-936b-74a7a5a2b392-istio-podinfo\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w\" (UID: \"f26d33aa-399c-46f8-936b-74a7a5a2b392\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"
Oct 30 00:47:20.617495 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.617399    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f26d33aa-399c-46f8-936b-74a7a5a2b392-istio-token\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w\" (UID: \"f26d33aa-399c-46f8-936b-74a7a5a2b392\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"
Oct 30 00:47:20.617495 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.617470    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26d33aa-399c-46f8-936b-74a7a5a2b392-istio-data\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w\" (UID: \"f26d33aa-399c-46f8-936b-74a7a5a2b392\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"
Oct 30 00:47:20.617628 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.617584    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k64gv\" (UniqueName: \"kubernetes.io/projected/f26d33aa-399c-46f8-936b-74a7a5a2b392-kube-api-access-k64gv\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w\" (UID: \"f26d33aa-399c-46f8-936b-74a7a5a2b392\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"
Oct 30 00:47:20.617703 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.617672    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26d33aa-399c-46f8-936b-74a7a5a2b392-workload-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w\" (UID: \"f26d33aa-399c-46f8-936b-74a7a5a2b392\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"
Oct 30 00:47:20.617811 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.617763    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26d33aa-399c-46f8-936b-74a7a5a2b392-credential-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w\" (UID: \"f26d33aa-399c-46f8-936b-74a7a5a2b392\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"
Oct 30 00:47:20.617857 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.617821    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26d33aa-399c-46f8-936b-74a7a5a2b392-workload-certs\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w\" (UID: \"f26d33aa-399c-46f8-936b-74a7a5a2b392\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"
Oct 30 00:47:20.617892 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.617855    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26d33aa-399c-46f8-936b-74a7a5a2b392-istio-envoy\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w\" (UID: \"f26d33aa-399c-46f8-936b-74a7a5a2b392\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"
Oct 30 00:47:20.617892 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.617870    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26d33aa-399c-46f8-936b-74a7a5a2b392-istiod-ca-cert\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w\" (UID: \"f26d33aa-399c-46f8-936b-74a7a5a2b392\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"
Oct 30 00:47:20.718612 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.718576    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26d33aa-399c-46f8-936b-74a7a5a2b392-credential-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w\" (UID: \"f26d33aa-399c-46f8-936b-74a7a5a2b392\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"
Oct 30 00:47:20.718612 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.718620    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26d33aa-399c-46f8-936b-74a7a5a2b392-workload-certs\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w\" (UID: \"f26d33aa-399c-46f8-936b-74a7a5a2b392\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"
Oct 30 00:47:20.718612 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.718667    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26d33aa-399c-46f8-936b-74a7a5a2b392-istio-envoy\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w\" (UID: \"f26d33aa-399c-46f8-936b-74a7a5a2b392\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"
Oct 30 00:47:20.719990 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.718690    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26d33aa-399c-46f8-936b-74a7a5a2b392-istiod-ca-cert\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w\" (UID: \"f26d33aa-399c-46f8-936b-74a7a5a2b392\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"
Oct 30 00:47:20.719990 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.718735    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26d33aa-399c-46f8-936b-74a7a5a2b392-istio-podinfo\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w\" (UID: \"f26d33aa-399c-46f8-936b-74a7a5a2b392\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"
Oct 30 00:47:20.719990 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.718827    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f26d33aa-399c-46f8-936b-74a7a5a2b392-istio-token\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w\" (UID: \"f26d33aa-399c-46f8-936b-74a7a5a2b392\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"
Oct 30 00:47:20.719990 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.718898    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26d33aa-399c-46f8-936b-74a7a5a2b392-istio-data\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w\" (UID: \"f26d33aa-399c-46f8-936b-74a7a5a2b392\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"
Oct 30 00:47:20.719990 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.719055    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26d33aa-399c-46f8-936b-74a7a5a2b392-credential-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w\" (UID: \"f26d33aa-399c-46f8-936b-74a7a5a2b392\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"
Oct 30 00:47:20.719990 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.719316    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k64gv\" (UniqueName: \"kubernetes.io/projected/f26d33aa-399c-46f8-936b-74a7a5a2b392-kube-api-access-k64gv\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w\" (UID: \"f26d33aa-399c-46f8-936b-74a7a5a2b392\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"
Oct 30 00:47:20.719990 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.719501    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26d33aa-399c-46f8-936b-74a7a5a2b392-workload-certs\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w\" (UID: \"f26d33aa-399c-46f8-936b-74a7a5a2b392\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"
Oct 30 00:47:20.719990 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.719545    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26d33aa-399c-46f8-936b-74a7a5a2b392-workload-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w\" (UID: \"f26d33aa-399c-46f8-936b-74a7a5a2b392\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"
Oct 30 00:47:20.719990 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.719670    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26d33aa-399c-46f8-936b-74a7a5a2b392-istio-data\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w\" (UID: \"f26d33aa-399c-46f8-936b-74a7a5a2b392\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"
Oct 30 00:47:20.719990 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.719682    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26d33aa-399c-46f8-936b-74a7a5a2b392-istiod-ca-cert\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w\" (UID: \"f26d33aa-399c-46f8-936b-74a7a5a2b392\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"
Oct 30 00:47:20.719990 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.719940    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26d33aa-399c-46f8-936b-74a7a5a2b392-workload-socket\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w\" (UID: \"f26d33aa-399c-46f8-936b-74a7a5a2b392\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"
Oct 30 00:47:20.721993 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.721968    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26d33aa-399c-46f8-936b-74a7a5a2b392-istio-podinfo\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w\" (UID: \"f26d33aa-399c-46f8-936b-74a7a5a2b392\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"
Oct 30 00:47:20.722296 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.722277    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26d33aa-399c-46f8-936b-74a7a5a2b392-istio-envoy\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w\" (UID: \"f26d33aa-399c-46f8-936b-74a7a5a2b392\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"
Oct 30 00:47:20.724556 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.724532    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"]
Oct 30 00:47:20.738663 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.738590    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"
Oct 30 00:47:20.750285 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.750251    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f26d33aa-399c-46f8-936b-74a7a5a2b392-istio-token\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w\" (UID: \"f26d33aa-399c-46f8-936b-74a7a5a2b392\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"
Oct 30 00:47:20.754578 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.754542    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k64gv\" (UniqueName: \"kubernetes.io/projected/f26d33aa-399c-46f8-936b-74a7a5a2b392-kube-api-access-k64gv\") pod \"e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w\" (UID: \"f26d33aa-399c-46f8-936b-74a7a5a2b392\") " pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"
Oct 30 00:47:20.755053 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.755029    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"]
Oct 30 00:47:20.813581 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.813552    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"
Oct 30 00:47:20.821304 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.821275    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eead8403-b3bd-4a6e-bfc0-0c1785eae339-istio-data\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm\" (UID: \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"
Oct 30 00:47:20.821426 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.821390    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eead8403-b3bd-4a6e-bfc0-0c1785eae339-workload-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm\" (UID: \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"
Oct 30 00:47:20.821486 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.821460    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eead8403-b3bd-4a6e-bfc0-0c1785eae339-credential-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm\" (UID: \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"
Oct 30 00:47:20.821613 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.821590    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eead8403-b3bd-4a6e-bfc0-0c1785eae339-istio-envoy\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm\" (UID: \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"
Oct 30 00:47:20.821810 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.821765    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eead8403-b3bd-4a6e-bfc0-0c1785eae339-workload-certs\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm\" (UID: \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"
Oct 30 00:47:20.821886 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.821812    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eead8403-b3bd-4a6e-bfc0-0c1785eae339-istiod-ca-cert\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm\" (UID: \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"
Oct 30 00:47:20.821886 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.821836    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tpnl\" (UniqueName: \"kubernetes.io/projected/eead8403-b3bd-4a6e-bfc0-0c1785eae339-kube-api-access-4tpnl\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm\" (UID: \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"
Oct 30 00:47:20.822006 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.821915    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eead8403-b3bd-4a6e-bfc0-0c1785eae339-istio-podinfo\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm\" (UID: \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"
Oct 30 00:47:20.822006 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.821977    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eead8403-b3bd-4a6e-bfc0-0c1785eae339-istio-token\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm\" (UID: \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"
Oct 30 00:47:20.836464 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.836438    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"]
Oct 30 00:47:20.842555 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.842531    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"
Oct 30 00:47:20.877754 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.877718    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"]
Oct 30 00:47:20.923043 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.923012    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eead8403-b3bd-4a6e-bfc0-0c1785eae339-istio-envoy\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm\" (UID: \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"
Oct 30 00:47:20.923178 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.923057    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d70acab5-9907-49ce-84aa-0b41757e7ae0-istio-token\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv\" (UID: \"d70acab5-9907-49ce-84aa-0b41757e7ae0\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"
Oct 30 00:47:20.923178 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.923109    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eead8403-b3bd-4a6e-bfc0-0c1785eae339-workload-certs\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm\" (UID: \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"
Oct 30 00:47:20.923178 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.923145    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eead8403-b3bd-4a6e-bfc0-0c1785eae339-istiod-ca-cert\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm\" (UID: \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"
Oct 30 00:47:20.923178 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.923166    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tpnl\" (UniqueName: \"kubernetes.io/projected/eead8403-b3bd-4a6e-bfc0-0c1785eae339-kube-api-access-4tpnl\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm\" (UID: \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"
Oct 30 00:47:20.923435 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.923192    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d70acab5-9907-49ce-84aa-0b41757e7ae0-istio-data\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv\" (UID: \"d70acab5-9907-49ce-84aa-0b41757e7ae0\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"
Oct 30 00:47:20.923435 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.923227    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gft2\" (UniqueName: \"kubernetes.io/projected/d70acab5-9907-49ce-84aa-0b41757e7ae0-kube-api-access-6gft2\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv\" (UID: \"d70acab5-9907-49ce-84aa-0b41757e7ae0\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"
Oct 30 00:47:20.923435 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.923253    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eead8403-b3bd-4a6e-bfc0-0c1785eae339-istio-podinfo\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm\" (UID: \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"
Oct 30 00:47:20.923435 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.923308    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d70acab5-9907-49ce-84aa-0b41757e7ae0-workload-certs\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv\" (UID: \"d70acab5-9907-49ce-84aa-0b41757e7ae0\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"
Oct 30 00:47:20.923435 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.923336    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d70acab5-9907-49ce-84aa-0b41757e7ae0-istio-envoy\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv\" (UID: \"d70acab5-9907-49ce-84aa-0b41757e7ae0\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"
Oct 30 00:47:20.923435 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.923356    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eead8403-b3bd-4a6e-bfc0-0c1785eae339-istio-token\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm\" (UID: \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"
Oct 30 00:47:20.923435 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.923373    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d70acab5-9907-49ce-84aa-0b41757e7ae0-istio-podinfo\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv\" (UID: \"d70acab5-9907-49ce-84aa-0b41757e7ae0\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"
Oct 30 00:47:20.923435 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.923400    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70acab5-9907-49ce-84aa-0b41757e7ae0-workload-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv\" (UID: \"d70acab5-9907-49ce-84aa-0b41757e7ae0\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"
Oct 30 00:47:20.923929 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.923437    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eead8403-b3bd-4a6e-bfc0-0c1785eae339-istio-data\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm\" (UID: \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"
Oct 30 00:47:20.923929 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.923486    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eead8403-b3bd-4a6e-bfc0-0c1785eae339-workload-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm\" (UID: \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"
Oct 30 00:47:20.923929 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.923504    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d70acab5-9907-49ce-84aa-0b41757e7ae0-istiod-ca-cert\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv\" (UID: \"d70acab5-9907-49ce-84aa-0b41757e7ae0\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"
Oct 30 00:47:20.923929 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.923563    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eead8403-b3bd-4a6e-bfc0-0c1785eae339-credential-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm\" (UID: \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"
Oct 30 00:47:20.923929 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.923581    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70acab5-9907-49ce-84aa-0b41757e7ae0-credential-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv\" (UID: \"d70acab5-9907-49ce-84aa-0b41757e7ae0\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"
Oct 30 00:47:20.926847 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.924264    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eead8403-b3bd-4a6e-bfc0-0c1785eae339-istio-data\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm\" (UID: \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"
Oct 30 00:47:20.926847 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.924508    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eead8403-b3bd-4a6e-bfc0-0c1785eae339-workload-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm\" (UID: \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"
Oct 30 00:47:20.926847 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.924636    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eead8403-b3bd-4a6e-bfc0-0c1785eae339-workload-certs\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm\" (UID: \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"
Oct 30 00:47:20.926847 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.924711    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eead8403-b3bd-4a6e-bfc0-0c1785eae339-credential-socket\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm\" (UID: \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"
Oct 30 00:47:20.926847 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.925512    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eead8403-b3bd-4a6e-bfc0-0c1785eae339-istiod-ca-cert\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm\" (UID: \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"
Oct 30 00:47:20.927975 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.927951    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eead8403-b3bd-4a6e-bfc0-0c1785eae339-istio-podinfo\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm\" (UID: \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"
Oct 30 00:47:20.928641 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.928609    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eead8403-b3bd-4a6e-bfc0-0c1785eae339-istio-envoy\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm\" (UID: \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"
Oct 30 00:47:20.946989 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.946915    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eead8403-b3bd-4a6e-bfc0-0c1785eae339-istio-token\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm\" (UID: \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"
Oct 30 00:47:20.947520 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:20.947498    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tpnl\" (UniqueName: \"kubernetes.io/projected/eead8403-b3bd-4a6e-bfc0-0c1785eae339-kube-api-access-4tpnl\") pod \"e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm\" (UID: \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\") " pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"
Oct 30 00:47:21.024844 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.024749    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d70acab5-9907-49ce-84aa-0b41757e7ae0-istio-data\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv\" (UID: \"d70acab5-9907-49ce-84aa-0b41757e7ae0\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"
Oct 30 00:47:21.024844 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.024803    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gft2\" (UniqueName: \"kubernetes.io/projected/d70acab5-9907-49ce-84aa-0b41757e7ae0-kube-api-access-6gft2\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv\" (UID: \"d70acab5-9907-49ce-84aa-0b41757e7ae0\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"
Oct 30 00:47:21.025101 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.024855    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d70acab5-9907-49ce-84aa-0b41757e7ae0-workload-certs\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv\" (UID: \"d70acab5-9907-49ce-84aa-0b41757e7ae0\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"
Oct 30 00:47:21.025101 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.024869    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d70acab5-9907-49ce-84aa-0b41757e7ae0-istio-envoy\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv\" (UID: \"d70acab5-9907-49ce-84aa-0b41757e7ae0\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"
Oct 30 00:47:21.025101 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.024946    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d70acab5-9907-49ce-84aa-0b41757e7ae0-istio-podinfo\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv\" (UID: \"d70acab5-9907-49ce-84aa-0b41757e7ae0\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"
Oct 30 00:47:21.025101 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.024986    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70acab5-9907-49ce-84aa-0b41757e7ae0-workload-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv\" (UID: \"d70acab5-9907-49ce-84aa-0b41757e7ae0\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"
Oct 30 00:47:21.025319 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.025256    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d70acab5-9907-49ce-84aa-0b41757e7ae0-istio-data\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv\" (UID: \"d70acab5-9907-49ce-84aa-0b41757e7ae0\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"
Oct 30 00:47:21.025373 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.025351    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d70acab5-9907-49ce-84aa-0b41757e7ae0-workload-certs\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv\" (UID: \"d70acab5-9907-49ce-84aa-0b41757e7ae0\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"
Oct 30 00:47:21.025491 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.025472    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d70acab5-9907-49ce-84aa-0b41757e7ae0-istiod-ca-cert\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv\" (UID: \"d70acab5-9907-49ce-84aa-0b41757e7ae0\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"
Oct 30 00:47:21.025588 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.025553    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70acab5-9907-49ce-84aa-0b41757e7ae0-workload-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv\" (UID: \"d70acab5-9907-49ce-84aa-0b41757e7ae0\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"
Oct 30 00:47:21.025644 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.025565    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70acab5-9907-49ce-84aa-0b41757e7ae0-credential-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv\" (UID: \"d70acab5-9907-49ce-84aa-0b41757e7ae0\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"
Oct 30 00:47:21.025700 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.025671    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d70acab5-9907-49ce-84aa-0b41757e7ae0-istio-token\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv\" (UID: \"d70acab5-9907-49ce-84aa-0b41757e7ae0\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"
Oct 30 00:47:21.026134 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.026091    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70acab5-9907-49ce-84aa-0b41757e7ae0-credential-socket\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv\" (UID: \"d70acab5-9907-49ce-84aa-0b41757e7ae0\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"
Oct 30 00:47:21.026265 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.026230    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d70acab5-9907-49ce-84aa-0b41757e7ae0-istiod-ca-cert\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv\" (UID: \"d70acab5-9907-49ce-84aa-0b41757e7ae0\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"
Oct 30 00:47:21.027429 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.027408    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d70acab5-9907-49ce-84aa-0b41757e7ae0-istio-envoy\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv\" (UID: \"d70acab5-9907-49ce-84aa-0b41757e7ae0\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"
Oct 30 00:47:21.027697 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.027680    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d70acab5-9907-49ce-84aa-0b41757e7ae0-istio-podinfo\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv\" (UID: \"d70acab5-9907-49ce-84aa-0b41757e7ae0\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"
Oct 30 00:47:21.047553 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.047528    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gft2\" (UniqueName: \"kubernetes.io/projected/d70acab5-9907-49ce-84aa-0b41757e7ae0-kube-api-access-6gft2\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv\" (UID: \"d70acab5-9907-49ce-84aa-0b41757e7ae0\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"
Oct 30 00:47:21.047836 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.047818    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d70acab5-9907-49ce-84aa-0b41757e7ae0-istio-token\") pod \"e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv\" (UID: \"d70acab5-9907-49ce-84aa-0b41757e7ae0\") " pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"
Oct 30 00:47:21.090349 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.090319    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"
Oct 30 00:47:21.145981 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.145944    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"]
Oct 30 00:47:21.155945 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:47:21.155921    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf26d33aa_399c_46f8_936b_74a7a5a2b392.slice/crio-f2d3471d7e63ab5f301a66e85a59bca609225d51a73073ebf974d7486e6d4d39 WatchSource:0}: Error finding container f2d3471d7e63ab5f301a66e85a59bca609225d51a73073ebf974d7486e6d4d39: Status 404 returned error can't find the container with id f2d3471d7e63ab5f301a66e85a59bca609225d51a73073ebf974d7486e6d4d39
Oct 30 00:47:21.159570 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.159268    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:47:21.159570 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.159356    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:47:21.159570 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.159403    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:47:21.175549 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.175382    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"
Oct 30 00:47:21.405591 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.405558    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"]
Oct 30 00:47:21.410311 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:47:21.410268    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeead8403_b3bd_4a6e_bfc0_0c1785eae339.slice/crio-bb443a70504bf068a78b7b6b0a41e080c6b15af549c350df987f3bf3940968e9 WatchSource:0}: Error finding container bb443a70504bf068a78b7b6b0a41e080c6b15af549c350df987f3bf3940968e9: Status 404 returned error can't find the container with id bb443a70504bf068a78b7b6b0a41e080c6b15af549c350df987f3bf3940968e9
Oct 30 00:47:21.413507 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.413470    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:47:21.413595 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.413567    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:47:21.413634 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.413613    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:47:21.535383 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.535175    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm" event={"ID":"eead8403-b3bd-4a6e-bfc0-0c1785eae339","Type":"ContainerStarted","Data":"7d5e90364f94a15744f1612d3b0c3b402ad4aa16e75c3d21687784f34c0f0bad"}
Oct 30 00:47:21.538958 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.537215    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm" event={"ID":"eead8403-b3bd-4a6e-bfc0-0c1785eae339","Type":"ContainerStarted","Data":"bb443a70504bf068a78b7b6b0a41e080c6b15af549c350df987f3bf3940968e9"}
Oct 30 00:47:21.543722 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.543586    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"]
Oct 30 00:47:21.555300 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.555255    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:47:21.555540 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.555518    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:47:21.555684 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.555664    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:47:21.604168 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.604133    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w" event={"ID":"f26d33aa-399c-46f8-936b-74a7a5a2b392","Type":"ContainerStarted","Data":"15b95fad515dc7fb206397f0d8097d932d188272302a91faeac9f774fbb0be80"}
Oct 30 00:47:21.604325 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.604191    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w" event={"ID":"f26d33aa-399c-46f8-936b-74a7a5a2b392","Type":"ContainerStarted","Data":"f2d3471d7e63ab5f301a66e85a59bca609225d51a73073ebf974d7486e6d4d39"}
Oct 30 00:47:21.628225 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.628167    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm" podStartSLOduration=1.628147153 podStartE2EDuration="1.628147153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:21.625616348 +0000 UTC m=+932.013292738" watchObservedRunningTime="2026-10-30 00:47:21.628147153 +0000 UTC m=+932.015823543"
Oct 30 00:47:21.671757 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.671686    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w" podStartSLOduration=1.671667886 podStartE2EDuration="1.671667886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:21.669958357 +0000 UTC m=+932.057634742" watchObservedRunningTime="2026-10-30 00:47:21.671667886 +0000 UTC m=+932.059344278"
Oct 30 00:47:21.815059 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:21.814445    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"
Oct 30 00:47:22.090612 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:22.090530    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"
Oct 30 00:47:22.092230 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:22.092198    4393 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 00:47:22.092367 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:22.092259    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm" podUID="eead8403-b3bd-4a6e-bfc0-0c1785eae339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 00:47:22.667815 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:22.667763    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv" event={"ID":"d70acab5-9907-49ce-84aa-0b41757e7ae0","Type":"ContainerStarted","Data":"5b5e79faf14232708e926597ebe7d3ca8f765d3925aacf89c685134c2ada02a4"}
Oct 30 00:47:22.668437 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:22.668405    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv" event={"ID":"d70acab5-9907-49ce-84aa-0b41757e7ae0","Type":"ContainerStarted","Data":"b9e419567cc1ac37f18ee9bfd6c604ce9b4af7f670b8a388b634288a2d7027c6"}
Oct 30 00:47:22.840802 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:22.831983    4393 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 00:47:22.840802 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:22.832060    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w" podUID="f26d33aa-399c-46f8-936b-74a7a5a2b392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 00:47:23.091452 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:23.091351    4393 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 00:47:23.091452 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:23.091411    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm" podUID="eead8403-b3bd-4a6e-bfc0-0c1785eae339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 00:47:23.178926 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:23.178877    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"
Oct 30 00:47:23.179227 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:23.178963    4393 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 00:47:23.179398 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:23.179368    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv" podUID="d70acab5-9907-49ce-84aa-0b41757e7ae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 00:47:23.824805 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:23.824202    4393 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 00:47:23.824805 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:23.824277    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w" podUID="f26d33aa-399c-46f8-936b-74a7a5a2b392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 00:47:24.090676 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:24.090587    4393 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 00:47:24.090913 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:24.090890    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm" podUID="eead8403-b3bd-4a6e-bfc0-0c1785eae339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 00:47:24.177806 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:24.177355    4393 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 00:47:24.177806 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:24.177418    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv" podUID="d70acab5-9907-49ce-84aa-0b41757e7ae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 00:47:24.841942 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:24.841905    4393 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 00:47:24.842619 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:24.842582    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w" podUID="f26d33aa-399c-46f8-936b-74a7a5a2b392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 00:47:25.091120 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:25.091077    4393 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 00:47:25.091312 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:25.091133    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm" podUID="eead8403-b3bd-4a6e-bfc0-0c1785eae339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 00:47:25.181805 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:25.176159    4393 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 00:47:25.181805 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:25.176220    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv" podUID="d70acab5-9907-49ce-84aa-0b41757e7ae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 00:47:25.913808 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:25.886035    4393 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 00:47:25.913808 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:25.886106    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w" podUID="f26d33aa-399c-46f8-936b-74a7a5a2b392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 00:47:26.121805 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:26.097079    4393 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 00:47:26.125345 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:26.125301    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm" podUID="eead8403-b3bd-4a6e-bfc0-0c1785eae339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 00:47:26.199022 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:26.198925    4393 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 00:47:26.199192 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:26.199020    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv" podUID="d70acab5-9907-49ce-84aa-0b41757e7ae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 00:47:26.588900 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:47:26.588768    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:47:26.832804 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:26.826924    4393 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 00:47:26.832804 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:26.827007    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w" podUID="f26d33aa-399c-46f8-936b-74a7a5a2b392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 00:47:27.096179 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:27.096074    4393 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 00:47:27.096748 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:27.096710    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm" podUID="eead8403-b3bd-4a6e-bfc0-0c1785eae339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 00:47:27.182830 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:27.175989    4393 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 00:47:27.182830 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:27.176092    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv" podUID="d70acab5-9907-49ce-84aa-0b41757e7ae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 00:47:27.841807 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:27.839162    4393 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 00:47:27.841807 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:27.839219    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w" podUID="f26d33aa-399c-46f8-936b-74a7a5a2b392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 00:47:28.097619 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:28.097536    4393 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 00:47:28.098186 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:28.098155    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm" podUID="eead8403-b3bd-4a6e-bfc0-0c1785eae339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 00:47:28.198129 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:28.198091    4393 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 00:47:28.198653 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:28.198604    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv" podUID="d70acab5-9907-49ce-84aa-0b41757e7ae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 00:47:28.816845 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:28.815898    4393 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 00:47:28.816845 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:28.815955    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w" podUID="f26d33aa-399c-46f8-936b-74a7a5a2b392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 00:47:29.092330 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:29.092235    4393 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 00:47:29.092639 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:29.092604    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm" podUID="eead8403-b3bd-4a6e-bfc0-0c1785eae339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 00:47:29.184639 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:29.184602    4393 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 00:47:29.185262 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:29.185227    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv" podUID="d70acab5-9907-49ce-84aa-0b41757e7ae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 00:47:29.816434 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:29.815334    4393 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 00:47:29.816434 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:29.815395    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w" podUID="f26d33aa-399c-46f8-936b-74a7a5a2b392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 00:47:30.105631 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:30.105163    4393 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 00:47:30.105631 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:30.105221    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm" podUID="eead8403-b3bd-4a6e-bfc0-0c1785eae339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 00:47:31.049046 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:31.048942    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g" podUID="fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 30 00:47:31.233804 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:31.233122    4393 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:47:31.233804 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:31.233229    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv" podUID="d70acab5-9907-49ce-84aa-0b41757e7ae0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:31.352999 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:31.352901    4393 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:47:31.353271 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:31.353241    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg" podUID="a8bdca6c-102d-4aaa-ab76-16a9123d6a74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:31.378006 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:31.377962    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x" podUID="28f3e4ff-5875-4e5d-8cab-60d1a5ba97be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:31.857841 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:31.856555    4393 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:47:31.857841 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:31.856636    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w" podUID="f26d33aa-399c-46f8-936b-74a7a5a2b392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:32.066234 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:32.052367    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g" podUID="fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:32.068537 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:32.068476    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"
Oct 30 00:47:32.094990 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:32.094959    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"
Oct 30 00:47:32.095945 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:32.095915    4393 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:47:32.111890 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:32.096181    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm" podUID="eead8403-b3bd-4a6e-bfc0-0c1785eae339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:32.115823 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:32.097373    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"
Oct 30 00:47:32.116045 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:32.116029    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"
Oct 30 00:47:32.116251 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:32.116240    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"
Oct 30 00:47:32.142509 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:32.142174    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv" podStartSLOduration=12.142154239 podStartE2EDuration="12.142154239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:22.741931241 +0000 UTC m=+933.129607634" watchObservedRunningTime="2026-10-30 00:47:32.142154239 +0000 UTC m=+942.529830629"
Oct 30 00:47:32.858071 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:32.857498    4393 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:47:32.858071 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:32.857580    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w" podUID="f26d33aa-399c-46f8-936b-74a7a5a2b392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:33.164010 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:33.163959    4393 patch_prober.go:29] interesting pod/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:47:33.164644 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:33.164594    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm" podUID="eead8403-b3bd-4a6e-bfc0-0c1785eae339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:33.680165 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:33.680133    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"
Oct 30 00:47:33.681497 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:33.681469    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"
Oct 30 00:47:33.682937 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:33.682909    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"
Oct 30 00:47:33.753952 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:33.753923    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"
Oct 30 00:47:33.754484 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:33.754437    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"
Oct 30 00:47:33.758077 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:33.758053    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"
Oct 30 00:47:44.100860 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:44.098871    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g" podUID="fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:44.389022 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:44.388972    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x" podUID="28f3e4ff-5875-4e5d-8cab-60d1a5ba97be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Oct 30 00:47:45.083903 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:45.083761    4393 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:47:45.084112 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:45.083895    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd" podUID="394e8f0b-d96a-44ec-9cdf-a1266ee49a91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:47:51.024385 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:51.024350    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"]
Oct 30 00:47:51.029010 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:51.028992    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"
Oct 30 00:47:51.062994 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:51.062960    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"]
Oct 30 00:47:51.087720 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:51.087683    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b626471-54fd-4301-89fa-cc1738c35cce-istio-data\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6\" (UID: \"3b626471-54fd-4301-89fa-cc1738c35cce\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"
Oct 30 00:47:51.087906 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:51.087733    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhrzq\" (UniqueName: \"kubernetes.io/projected/3b626471-54fd-4301-89fa-cc1738c35cce-kube-api-access-jhrzq\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6\" (UID: \"3b626471-54fd-4301-89fa-cc1738c35cce\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"
Oct 30 00:47:51.087906 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:51.087873    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b626471-54fd-4301-89fa-cc1738c35cce-workload-certs\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6\" (UID: \"3b626471-54fd-4301-89fa-cc1738c35cce\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"
Oct 30 00:47:51.088029 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:51.087928    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b626471-54fd-4301-89fa-cc1738c35cce-credential-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6\" (UID: \"3b626471-54fd-4301-89fa-cc1738c35cce\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"
Oct 30 00:47:51.088091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:51.088042    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b626471-54fd-4301-89fa-cc1738c35cce-istio-token\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6\" (UID: \"3b626471-54fd-4301-89fa-cc1738c35cce\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"
Oct 30 00:47:51.088186 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:51.088166    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b626471-54fd-4301-89fa-cc1738c35cce-istio-envoy\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6\" (UID: \"3b626471-54fd-4301-89fa-cc1738c35cce\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"
Oct 30 00:47:51.088224 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:51.088197    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b626471-54fd-4301-89fa-cc1738c35cce-istio-podinfo\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6\" (UID: \"3b626471-54fd-4301-89fa-cc1738c35cce\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"
Oct 30 00:47:51.088263 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:51.088227    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b626471-54fd-4301-89fa-cc1738c35cce-workload-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6\" (UID: \"3b626471-54fd-4301-89fa-cc1738c35cce\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"
Oct 30 00:47:51.088263 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:51.088242    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b626471-54fd-4301-89fa-cc1738c35cce-istiod-ca-cert\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6\" (UID: \"3b626471-54fd-4301-89fa-cc1738c35cce\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"
Oct 30 00:47:51.189296 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:51.189256    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b626471-54fd-4301-89fa-cc1738c35cce-istio-envoy\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6\" (UID: \"3b626471-54fd-4301-89fa-cc1738c35cce\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"
Oct 30 00:47:51.189296 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:51.189297    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b626471-54fd-4301-89fa-cc1738c35cce-istio-podinfo\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6\" (UID: \"3b626471-54fd-4301-89fa-cc1738c35cce\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"
Oct 30 00:47:51.189606 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:51.189335    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b626471-54fd-4301-89fa-cc1738c35cce-workload-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6\" (UID: \"3b626471-54fd-4301-89fa-cc1738c35cce\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"
Oct 30 00:47:51.189606 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:51.189357    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b626471-54fd-4301-89fa-cc1738c35cce-istiod-ca-cert\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6\" (UID: \"3b626471-54fd-4301-89fa-cc1738c35cce\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"
Oct 30 00:47:51.189606 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:51.189504    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b626471-54fd-4301-89fa-cc1738c35cce-istio-data\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6\" (UID: \"3b626471-54fd-4301-89fa-cc1738c35cce\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"
Oct 30 00:47:51.189606 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:51.189559    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhrzq\" (UniqueName: \"kubernetes.io/projected/3b626471-54fd-4301-89fa-cc1738c35cce-kube-api-access-jhrzq\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6\" (UID: \"3b626471-54fd-4301-89fa-cc1738c35cce\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"
Oct 30 00:47:51.189923 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:51.189682    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b626471-54fd-4301-89fa-cc1738c35cce-workload-certs\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6\" (UID: \"3b626471-54fd-4301-89fa-cc1738c35cce\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"
Oct 30 00:47:51.189923 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:51.189800    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b626471-54fd-4301-89fa-cc1738c35cce-credential-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6\" (UID: \"3b626471-54fd-4301-89fa-cc1738c35cce\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"
Oct 30 00:47:51.189923 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:51.189849    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b626471-54fd-4301-89fa-cc1738c35cce-workload-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6\" (UID: \"3b626471-54fd-4301-89fa-cc1738c35cce\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"
Oct 30 00:47:51.189923 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:51.189863    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b626471-54fd-4301-89fa-cc1738c35cce-istio-token\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6\" (UID: \"3b626471-54fd-4301-89fa-cc1738c35cce\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"
Oct 30 00:47:51.190140 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:51.190035    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b626471-54fd-4301-89fa-cc1738c35cce-workload-certs\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6\" (UID: \"3b626471-54fd-4301-89fa-cc1738c35cce\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"
Oct 30 00:47:51.190192 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:51.190152    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b626471-54fd-4301-89fa-cc1738c35cce-istiod-ca-cert\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6\" (UID: \"3b626471-54fd-4301-89fa-cc1738c35cce\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"
Oct 30 00:47:51.190402 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:51.190380    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b626471-54fd-4301-89fa-cc1738c35cce-istio-data\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6\" (UID: \"3b626471-54fd-4301-89fa-cc1738c35cce\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"
Oct 30 00:47:51.190440 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:51.190380    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b626471-54fd-4301-89fa-cc1738c35cce-credential-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6\" (UID: \"3b626471-54fd-4301-89fa-cc1738c35cce\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"
Oct 30 00:47:51.191877 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:51.191849    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b626471-54fd-4301-89fa-cc1738c35cce-istio-envoy\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6\" (UID: \"3b626471-54fd-4301-89fa-cc1738c35cce\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"
Oct 30 00:47:51.191980 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:51.191910    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b626471-54fd-4301-89fa-cc1738c35cce-istio-podinfo\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6\" (UID: \"3b626471-54fd-4301-89fa-cc1738c35cce\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"
Oct 30 00:47:51.212023 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:51.211995    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b626471-54fd-4301-89fa-cc1738c35cce-istio-token\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6\" (UID: \"3b626471-54fd-4301-89fa-cc1738c35cce\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"
Oct 30 00:47:51.212111 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:51.212093    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhrzq\" (UniqueName: \"kubernetes.io/projected/3b626471-54fd-4301-89fa-cc1738c35cce-kube-api-access-jhrzq\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6\" (UID: \"3b626471-54fd-4301-89fa-cc1738c35cce\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"
Oct 30 00:47:51.346119 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:51.346031    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"
Oct 30 00:47:51.668025 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:51.667993    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"]
Oct 30 00:47:51.668201 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:47:51.668112    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b626471_54fd_4301_89fa_cc1738c35cce.slice/crio-340e58de3e26202d8e20f6d1f10a6ae1abf8afc31c7340a47e77ffbae35b3715 WatchSource:0}: Error finding container 340e58de3e26202d8e20f6d1f10a6ae1abf8afc31c7340a47e77ffbae35b3715: Status 404 returned error can't find the container with id 340e58de3e26202d8e20f6d1f10a6ae1abf8afc31c7340a47e77ffbae35b3715
Oct 30 00:47:51.671054 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:51.671017    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:47:51.671139 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:51.671100    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:47:51.671179 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:51.671144    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:47:52.385733 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:52.385690    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6" event={"ID":"3b626471-54fd-4301-89fa-cc1738c35cce","Type":"ContainerStarted","Data":"f5ea02f1a8c1d0a0248db155606b648e531de07392eda5cdc00fc4edead745a8"}
Oct 30 00:47:52.386234 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:52.385743    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6" event={"ID":"3b626471-54fd-4301-89fa-cc1738c35cce","Type":"ContainerStarted","Data":"340e58de3e26202d8e20f6d1f10a6ae1abf8afc31c7340a47e77ffbae35b3715"}
Oct 30 00:47:52.454022 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:52.453967    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6" podStartSLOduration=2.4539504770000002 podStartE2EDuration="2.453950477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:47:52.441555422 +0000 UTC m=+962.829231813" watchObservedRunningTime="2026-10-30 00:47:52.453950477 +0000 UTC m=+962.841626866"
Oct 30 00:47:53.346928 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:53.346894    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"
Oct 30 00:47:53.348351 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:53.348326    4393 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 00:47:53.348496 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:53.348372    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6" podUID="3b626471-54fd-4301-89fa-cc1738c35cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 00:47:54.347336 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:54.347302    4393 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 00:47:54.347876 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:54.347358    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6" podUID="3b626471-54fd-4301-89fa-cc1738c35cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 00:47:55.346841 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:55.346807    4393 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 00:47:55.347019 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:55.346878    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6" podUID="3b626471-54fd-4301-89fa-cc1738c35cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 00:47:56.347188 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:56.347157    4393 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 00:47:56.347636 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:56.347216    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6" podUID="3b626471-54fd-4301-89fa-cc1738c35cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 00:47:56.533552 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:47:56.533518    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:47:57.346395 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:57.346359    4393 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 00:47:57.346598 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:57.346409    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6" podUID="3b626471-54fd-4301-89fa-cc1738c35cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 00:47:58.347275 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:58.347232    4393 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 00:47:58.347711 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:47:58.347330    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6" podUID="3b626471-54fd-4301-89fa-cc1738c35cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 00:48:00.460817 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:00.446928    4393 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:48:00.460817 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:00.447230    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6" podUID="3b626471-54fd-4301-89fa-cc1738c35cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:01.314346 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:01.313916    4393 patch_prober.go:29] interesting pod/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:48:01.314346 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:01.313988    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz" podUID="75776f35-e39d-4823-ac2a-61e5b4e69619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:01.377747 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:01.376921    4393 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:48:01.377747 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:01.376987    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg" podUID="a8bdca6c-102d-4aaa-ab76-16a9123d6a74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:01.490898 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:01.479405    4393 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:48:01.490898 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:01.479484    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6" podUID="3b626471-54fd-4301-89fa-cc1738c35cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:02.535169 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:02.524256    4393 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:48:02.535169 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:02.529895    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6" podUID="3b626471-54fd-4301-89fa-cc1738c35cce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:02.925646 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:02.925586    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"
Oct 30 00:48:02.926131 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:02.926110    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"
Oct 30 00:48:02.927220 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:02.927201    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"
Oct 30 00:48:23.130552 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:23.130502    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g" podUID="fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 30 00:48:23.385974 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:23.385873    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x" podUID="28f3e4ff-5875-4e5d-8cab-60d1a5ba97be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Oct 30 00:48:24.149926 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:24.149879    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g" podUID="fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 30 00:48:24.196807 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:24.167139    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"
Oct 30 00:48:24.431479 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:24.425874    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x" podUID="28f3e4ff-5875-4e5d-8cab-60d1a5ba97be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:24.461801 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:24.460873    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"
Oct 30 00:48:24.547035 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:24.546599    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"
Oct 30 00:48:24.552370 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:24.552344    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"
Oct 30 00:48:26.514150 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:48:26.514111    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:48:46.342806 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:46.341892    4393 patch_prober.go:29] interesting pod/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:48:46.342806 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:46.341962    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg" podUID="a8bdca6c-102d-4aaa-ab76-16a9123d6a74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:46.581980 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:46.581927    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g" podUID="fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:48:49.952266 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:49.952236    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q"]
Oct 30 00:48:49.957062 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:49.957043    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q"
Oct 30 00:48:49.967467 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:49.967438    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46b01388-7da7-4e63-95f3-afc811bed013-home\") pod \"distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q\" (UID: \"46b01388-7da7-4e63-95f3-afc811bed013\") " pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q"
Oct 30 00:48:49.967588 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:49.967506    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpkcg\" (UniqueName: \"kubernetes.io/projected/46b01388-7da7-4e63-95f3-afc811bed013-kube-api-access-mpkcg\") pod \"distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q\" (UID: \"46b01388-7da7-4e63-95f3-afc811bed013\") " pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q"
Oct 30 00:48:49.967636 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:49.967595    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46b01388-7da7-4e63-95f3-afc811bed013-dshm\") pod \"distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q\" (UID: \"46b01388-7da7-4e63-95f3-afc811bed013\") " pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q"
Oct 30 00:48:49.967680 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:49.967657    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46b01388-7da7-4e63-95f3-afc811bed013-model-cache\") pod \"distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q\" (UID: \"46b01388-7da7-4e63-95f3-afc811bed013\") " pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q"
Oct 30 00:48:49.967851 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:49.967817    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46b01388-7da7-4e63-95f3-afc811bed013-tmp-dir\") pod \"distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q\" (UID: \"46b01388-7da7-4e63-95f3-afc811bed013\") " pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q"
Oct 30 00:48:49.967969 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:49.967950    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46b01388-7da7-4e63-95f3-afc811bed013-tls-certs\") pod \"distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q\" (UID: \"46b01388-7da7-4e63-95f3-afc811bed013\") " pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q"
Oct 30 00:48:49.975256 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:49.975231    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q"]
Oct 30 00:48:49.982966 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:49.982942    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-cefb27\"/\"distinct-w3-cefb27-kserve-self-signed-certs\""
Oct 30 00:48:49.985456 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:49.985435    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-cefb27\"/\"openshift-service-ca.crt\""
Oct 30 00:48:49.985529 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:49.985457    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-cefb27\"/\"kube-root-ca.crt\""
Oct 30 00:48:49.985529 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:49.985478    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-cefb27\"/\"default-dockercfg-jzvk5\""
Oct 30 00:48:50.069424 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:50.069385    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46b01388-7da7-4e63-95f3-afc811bed013-tls-certs\") pod \"distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q\" (UID: \"46b01388-7da7-4e63-95f3-afc811bed013\") " pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q"
Oct 30 00:48:50.069605 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:50.069510    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46b01388-7da7-4e63-95f3-afc811bed013-home\") pod \"distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q\" (UID: \"46b01388-7da7-4e63-95f3-afc811bed013\") " pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q"
Oct 30 00:48:50.069605 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:50.069546    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpkcg\" (UniqueName: \"kubernetes.io/projected/46b01388-7da7-4e63-95f3-afc811bed013-kube-api-access-mpkcg\") pod \"distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q\" (UID: \"46b01388-7da7-4e63-95f3-afc811bed013\") " pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q"
Oct 30 00:48:50.069605 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:50.069583    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46b01388-7da7-4e63-95f3-afc811bed013-dshm\") pod \"distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q\" (UID: \"46b01388-7da7-4e63-95f3-afc811bed013\") " pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q"
Oct 30 00:48:50.069819 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:50.069626    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46b01388-7da7-4e63-95f3-afc811bed013-model-cache\") pod \"distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q\" (UID: \"46b01388-7da7-4e63-95f3-afc811bed013\") " pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q"
Oct 30 00:48:50.069819 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:50.069688    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46b01388-7da7-4e63-95f3-afc811bed013-tmp-dir\") pod \"distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q\" (UID: \"46b01388-7da7-4e63-95f3-afc811bed013\") " pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q"
Oct 30 00:48:50.070003 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:50.069982    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46b01388-7da7-4e63-95f3-afc811bed013-home\") pod \"distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q\" (UID: \"46b01388-7da7-4e63-95f3-afc811bed013\") " pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q"
Oct 30 00:48:50.070135 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:50.070114    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46b01388-7da7-4e63-95f3-afc811bed013-tmp-dir\") pod \"distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q\" (UID: \"46b01388-7da7-4e63-95f3-afc811bed013\") " pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q"
Oct 30 00:48:50.070358 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:50.070261    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46b01388-7da7-4e63-95f3-afc811bed013-model-cache\") pod \"distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q\" (UID: \"46b01388-7da7-4e63-95f3-afc811bed013\") " pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q"
Oct 30 00:48:50.071944 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:50.071921    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46b01388-7da7-4e63-95f3-afc811bed013-dshm\") pod \"distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q\" (UID: \"46b01388-7da7-4e63-95f3-afc811bed013\") " pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q"
Oct 30 00:48:50.072137 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:50.072120    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46b01388-7da7-4e63-95f3-afc811bed013-tls-certs\") pod \"distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q\" (UID: \"46b01388-7da7-4e63-95f3-afc811bed013\") " pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q"
Oct 30 00:48:50.092183 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:50.092159    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpkcg\" (UniqueName: \"kubernetes.io/projected/46b01388-7da7-4e63-95f3-afc811bed013-kube-api-access-mpkcg\") pod \"distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q\" (UID: \"46b01388-7da7-4e63-95f3-afc811bed013\") " pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q"
Oct 30 00:48:50.271414 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:50.271323    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q"
Oct 30 00:48:50.646232 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:50.646208    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q"]
Oct 30 00:48:50.647063 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:48:50.647032    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46b01388_7da7_4e63_95f3_afc811bed013.slice/crio-d7c571ef5698359dc40d2911163bab63e594662a066fdb86fc63487d03996497 WatchSource:0}: Error finding container d7c571ef5698359dc40d2911163bab63e594662a066fdb86fc63487d03996497: Status 404 returned error can't find the container with id d7c571ef5698359dc40d2911163bab63e594662a066fdb86fc63487d03996497
Oct 30 00:48:51.201325 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.201288    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"]
Oct 30 00:48:51.208476 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.208451    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"
Oct 30 00:48:51.231132 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.230244    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"]
Oct 30 00:48:51.231132 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.230812    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q" event={"ID":"46b01388-7da7-4e63-95f3-afc811bed013","Type":"ContainerStarted","Data":"99c0224efa3f8e76a5e54cd092be370ed6a0a6ef633643fcd7cf2ad75632246a"}
Oct 30 00:48:51.231132 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.230854    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q" event={"ID":"46b01388-7da7-4e63-95f3-afc811bed013","Type":"ContainerStarted","Data":"d7c571ef5698359dc40d2911163bab63e594662a066fdb86fc63487d03996497"}
Oct 30 00:48:51.272080 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.272051    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q"
Oct 30 00:48:51.296773 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.296721    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q" podStartSLOduration=2.296708595 podStartE2EDuration="2.296708595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:51.29399667 +0000 UTC m=+1021.681673060" watchObservedRunningTime="2026-10-30 00:48:51.296708595 +0000 UTC m=+1021.684384984"
Oct 30 00:48:51.307265 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.307239    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6806a61-7f2d-410f-adab-680fdc4a9b08-credential-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp\" (UID: \"b6806a61-7f2d-410f-adab-680fdc4a9b08\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"
Oct 30 00:48:51.307480 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.307458    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6806a61-7f2d-410f-adab-680fdc4a9b08-istio-data\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp\" (UID: \"b6806a61-7f2d-410f-adab-680fdc4a9b08\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"
Oct 30 00:48:51.307621 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.307493    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjglw\" (UniqueName: \"kubernetes.io/projected/b6806a61-7f2d-410f-adab-680fdc4a9b08-kube-api-access-wjglw\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp\" (UID: \"b6806a61-7f2d-410f-adab-680fdc4a9b08\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"
Oct 30 00:48:51.307621 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.307610    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6806a61-7f2d-410f-adab-680fdc4a9b08-istio-podinfo\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp\" (UID: \"b6806a61-7f2d-410f-adab-680fdc4a9b08\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"
Oct 30 00:48:51.307757 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.307645    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6806a61-7f2d-410f-adab-680fdc4a9b08-istiod-ca-cert\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp\" (UID: \"b6806a61-7f2d-410f-adab-680fdc4a9b08\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"
Oct 30 00:48:51.307757 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.307666    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6806a61-7f2d-410f-adab-680fdc4a9b08-istio-token\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp\" (UID: \"b6806a61-7f2d-410f-adab-680fdc4a9b08\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"
Oct 30 00:48:51.307757 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.307685    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6806a61-7f2d-410f-adab-680fdc4a9b08-workload-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp\" (UID: \"b6806a61-7f2d-410f-adab-680fdc4a9b08\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"
Oct 30 00:48:51.307757 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.307746    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6806a61-7f2d-410f-adab-680fdc4a9b08-istio-envoy\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp\" (UID: \"b6806a61-7f2d-410f-adab-680fdc4a9b08\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"
Oct 30 00:48:51.307987 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.307872    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6806a61-7f2d-410f-adab-680fdc4a9b08-workload-certs\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp\" (UID: \"b6806a61-7f2d-410f-adab-680fdc4a9b08\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"
Oct 30 00:48:51.438833 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.433703    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6806a61-7f2d-410f-adab-680fdc4a9b08-workload-certs\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp\" (UID: \"b6806a61-7f2d-410f-adab-680fdc4a9b08\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"
Oct 30 00:48:51.438833 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.433833    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6806a61-7f2d-410f-adab-680fdc4a9b08-credential-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp\" (UID: \"b6806a61-7f2d-410f-adab-680fdc4a9b08\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"
Oct 30 00:48:51.438833 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.433985    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6806a61-7f2d-410f-adab-680fdc4a9b08-istio-data\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp\" (UID: \"b6806a61-7f2d-410f-adab-680fdc4a9b08\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"
Oct 30 00:48:51.438833 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.434024    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjglw\" (UniqueName: \"kubernetes.io/projected/b6806a61-7f2d-410f-adab-680fdc4a9b08-kube-api-access-wjglw\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp\" (UID: \"b6806a61-7f2d-410f-adab-680fdc4a9b08\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"
Oct 30 00:48:51.438833 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.434105    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6806a61-7f2d-410f-adab-680fdc4a9b08-istio-podinfo\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp\" (UID: \"b6806a61-7f2d-410f-adab-680fdc4a9b08\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"
Oct 30 00:48:51.438833 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.434148    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6806a61-7f2d-410f-adab-680fdc4a9b08-istiod-ca-cert\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp\" (UID: \"b6806a61-7f2d-410f-adab-680fdc4a9b08\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"
Oct 30 00:48:51.438833 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.434181    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6806a61-7f2d-410f-adab-680fdc4a9b08-istio-token\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp\" (UID: \"b6806a61-7f2d-410f-adab-680fdc4a9b08\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"
Oct 30 00:48:51.438833 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.434209    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6806a61-7f2d-410f-adab-680fdc4a9b08-workload-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp\" (UID: \"b6806a61-7f2d-410f-adab-680fdc4a9b08\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"
Oct 30 00:48:51.438833 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.434255    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6806a61-7f2d-410f-adab-680fdc4a9b08-istio-envoy\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp\" (UID: \"b6806a61-7f2d-410f-adab-680fdc4a9b08\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"
Oct 30 00:48:51.438833 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.435981    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6806a61-7f2d-410f-adab-680fdc4a9b08-workload-certs\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp\" (UID: \"b6806a61-7f2d-410f-adab-680fdc4a9b08\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"
Oct 30 00:48:51.438833 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.436283    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6806a61-7f2d-410f-adab-680fdc4a9b08-credential-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp\" (UID: \"b6806a61-7f2d-410f-adab-680fdc4a9b08\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"
Oct 30 00:48:51.438833 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.436511    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6806a61-7f2d-410f-adab-680fdc4a9b08-istio-data\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp\" (UID: \"b6806a61-7f2d-410f-adab-680fdc4a9b08\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"
Oct 30 00:48:51.444472 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.440286    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6806a61-7f2d-410f-adab-680fdc4a9b08-istiod-ca-cert\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp\" (UID: \"b6806a61-7f2d-410f-adab-680fdc4a9b08\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"
Oct 30 00:48:51.444472 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.441370    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6806a61-7f2d-410f-adab-680fdc4a9b08-workload-socket\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp\" (UID: \"b6806a61-7f2d-410f-adab-680fdc4a9b08\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"
Oct 30 00:48:51.444472 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.443965    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6806a61-7f2d-410f-adab-680fdc4a9b08-istio-envoy\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp\" (UID: \"b6806a61-7f2d-410f-adab-680fdc4a9b08\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"
Oct 30 00:48:51.461204 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.460428    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6806a61-7f2d-410f-adab-680fdc4a9b08-istio-podinfo\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp\" (UID: \"b6806a61-7f2d-410f-adab-680fdc4a9b08\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"
Oct 30 00:48:51.468316 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.468282    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6806a61-7f2d-410f-adab-680fdc4a9b08-istio-token\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp\" (UID: \"b6806a61-7f2d-410f-adab-680fdc4a9b08\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"
Oct 30 00:48:51.474498 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.469120    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjglw\" (UniqueName: \"kubernetes.io/projected/b6806a61-7f2d-410f-adab-680fdc4a9b08-kube-api-access-wjglw\") pod \"e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp\" (UID: \"b6806a61-7f2d-410f-adab-680fdc4a9b08\") " pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"
Oct 30 00:48:51.529369 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.529333    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"
Oct 30 00:48:51.851109 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.851078    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"]
Oct 30 00:48:51.851452 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:48:51.851427    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6806a61_7f2d_410f_adab_680fdc4a9b08.slice/crio-c6446b96b94565729edf7f098c4128da4551005fad01c022e08c98c67fffe7eb WatchSource:0}: Error finding container c6446b96b94565729edf7f098c4128da4551005fad01c022e08c98c67fffe7eb: Status 404 returned error can't find the container with id c6446b96b94565729edf7f098c4128da4551005fad01c022e08c98c67fffe7eb
Oct 30 00:48:51.854032 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.853997    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:48:51.854110 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.854067    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:48:51.854110 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:51.854100    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:48:52.261803 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:52.261095    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp" event={"ID":"b6806a61-7f2d-410f-adab-680fdc4a9b08","Type":"ContainerStarted","Data":"de79eb65c4f6bc042c4cb5938ab390e46be7dfe47151123d9948e3239c2ca8e0"}
Oct 30 00:48:52.261803 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:52.261154    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp" event={"ID":"b6806a61-7f2d-410f-adab-680fdc4a9b08","Type":"ContainerStarted","Data":"c6446b96b94565729edf7f098c4128da4551005fad01c022e08c98c67fffe7eb"}
Oct 30 00:48:52.314478 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:52.308384    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp" podStartSLOduration=1.308369549 podStartE2EDuration="1.308369549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:48:52.307132314 +0000 UTC m=+1022.694808703" watchObservedRunningTime="2026-10-30 00:48:52.308369549 +0000 UTC m=+1022.696045939"
Oct 30 00:48:52.530203 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:52.530090    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"
Oct 30 00:48:52.531729 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:52.531702    4393 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 00:48:52.531887 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:52.531750    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp" podUID="b6806a61-7f2d-410f-adab-680fdc4a9b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 00:48:53.530221 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:53.530170    4393 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 00:48:53.530749 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:53.530244    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp" podUID="b6806a61-7f2d-410f-adab-680fdc4a9b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 00:48:54.131426 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:54.131391    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8804-7c671b"]
Oct 30 00:48:54.135838 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:54.135820    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8804-7c671b"
Oct 30 00:48:54.151146 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:54.151115    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8804-7c671b"]
Oct 30 00:48:54.156716 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:54.156689    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nrxkf\""
Oct 30 00:48:54.172246 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:54.172219    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vtch\" (UniqueName: \"kubernetes.io/projected/96bfc194-7371-4338-b143-64e6329e9469-kube-api-access-5vtch\") pod \"test-curl-8804-7c671b\" (UID: \"96bfc194-7371-4338-b143-64e6329e9469\") " pod="opendatahub/test-curl-8804-7c671b"
Oct 30 00:48:54.273213 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:54.273176    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vtch\" (UniqueName: \"kubernetes.io/projected/96bfc194-7371-4338-b143-64e6329e9469-kube-api-access-5vtch\") pod \"test-curl-8804-7c671b\" (UID: \"96bfc194-7371-4338-b143-64e6329e9469\") " pod="opendatahub/test-curl-8804-7c671b"
Oct 30 00:48:54.295815 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:54.295750    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vtch\" (UniqueName: \"kubernetes.io/projected/96bfc194-7371-4338-b143-64e6329e9469-kube-api-access-5vtch\") pod \"test-curl-8804-7c671b\" (UID: \"96bfc194-7371-4338-b143-64e6329e9469\") " pod="opendatahub/test-curl-8804-7c671b"
Oct 30 00:48:54.450383 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:54.450296    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8804-7c671b"
Oct 30 00:48:54.532987 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:54.532416    4393 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 00:48:54.532987 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:54.532480    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp" podUID="b6806a61-7f2d-410f-adab-680fdc4a9b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 00:48:54.771284 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:48:54.771250    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96bfc194_7371_4338_b143_64e6329e9469.slice/crio-d0c123ae3cf4d990f2d39ba25946c2e735ca2502ced32aa5e65f357fcec13cca WatchSource:0}: Error finding container d0c123ae3cf4d990f2d39ba25946c2e735ca2502ced32aa5e65f357fcec13cca: Status 404 returned error can't find the container with id d0c123ae3cf4d990f2d39ba25946c2e735ca2502ced32aa5e65f357fcec13cca
Oct 30 00:48:54.773739 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:54.773708    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8804-7c671b"]
Oct 30 00:48:54.773943 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:54.773922    4393 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:48:55.280144 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:55.280111    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8804-7c671b" event={"ID":"96bfc194-7371-4338-b143-64e6329e9469","Type":"ContainerStarted","Data":"d0c123ae3cf4d990f2d39ba25946c2e735ca2502ced32aa5e65f357fcec13cca"}
Oct 30 00:48:55.530113 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:55.530075    4393 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 00:48:55.530368 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:55.530145    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp" podUID="b6806a61-7f2d-410f-adab-680fdc4a9b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 00:48:56.530433 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:56.530400    4393 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 00:48:56.530934 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:56.530462    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp" podUID="b6806a61-7f2d-410f-adab-680fdc4a9b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 00:48:56.535516 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:48:56.535484    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:48:57.529822 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:57.529752    4393 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 00:48:57.530029 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:57.529828    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp" podUID="b6806a61-7f2d-410f-adab-680fdc4a9b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 00:48:58.529678 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:58.529637    4393 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 00:48:58.530151 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:58.529704    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp" podUID="b6806a61-7f2d-410f-adab-680fdc4a9b08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 00:48:59.534208 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:59.534177    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"
Oct 30 00:48:59.534694 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:59.534547    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"
Oct 30 00:48:59.535182 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:48:59.535160    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"
Oct 30 00:49:00.272142 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:00.272117    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q"
Oct 30 00:49:00.286910 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:00.286881    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q"
Oct 30 00:49:00.365031 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:00.365006    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q"
Oct 30 00:49:04.408967 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:04.408925    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8804-7c671b" event={"ID":"96bfc194-7371-4338-b143-64e6329e9469","Type":"ContainerStarted","Data":"de19130ac87b01bfa87177bb224431eccff8c0491a5656681128c1678e2c74ec"}
Oct 30 00:49:04.448933 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:04.448880    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8804-7c671b" podStartSLOduration=1.839705753 podStartE2EDuration="10.448866582s" totalImagesPullingTime="8.609160829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:48:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:04.445122519 +0000 UTC m=+1034.832798907" watchObservedRunningTime="2026-10-30 00:49:04.448866582 +0000 UTC m=+1034.836542970"
Oct 30 00:49:05.323598 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:05.323563    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8804-7c671b"]
Oct 30 00:49:05.327954 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:05.327925    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8804-7c671b"]
Oct 30 00:49:05.499814 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:05.499766    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8804-01fe52"]
Oct 30 00:49:05.500449 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:05.500434    4393 cpu_manager.go:414] "containerMap: removing container" podUID="96bfc194-7371-4338-b143-64e6329e9469" containerName="test-curl-8804-7c671b"
Oct 30 00:49:05.500501 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:05.500450    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96bfc194-7371-4338-b143-64e6329e9469" containerName="test-curl-8804-7c671b"
Oct 30 00:49:05.500585 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:05.500576    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="96bfc194-7371-4338-b143-64e6329e9469" containerName="test-curl-8804-7c671b"
Oct 30 00:49:05.504361 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:05.504342    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8804-01fe52"
Oct 30 00:49:05.521551 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:05.521519    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8804-01fe52"]
Oct 30 00:49:05.585523 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:05.585437    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c96kx\" (UniqueName: \"kubernetes.io/projected/233a53ca-0f95-49b1-86e0-bebf6591475b-kube-api-access-c96kx\") pod \"test-curl-8804-01fe52\" (UID: \"233a53ca-0f95-49b1-86e0-bebf6591475b\") " pod="opendatahub/test-curl-8804-01fe52"
Oct 30 00:49:05.688091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:05.688046    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c96kx\" (UniqueName: \"kubernetes.io/projected/233a53ca-0f95-49b1-86e0-bebf6591475b-kube-api-access-c96kx\") pod \"test-curl-8804-01fe52\" (UID: \"233a53ca-0f95-49b1-86e0-bebf6591475b\") " pod="opendatahub/test-curl-8804-01fe52"
Oct 30 00:49:05.750804 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:05.720800    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c96kx\" (UniqueName: \"kubernetes.io/projected/233a53ca-0f95-49b1-86e0-bebf6591475b-kube-api-access-c96kx\") pod \"test-curl-8804-01fe52\" (UID: \"233a53ca-0f95-49b1-86e0-bebf6591475b\") " pod="opendatahub/test-curl-8804-01fe52"
Oct 30 00:49:05.819544 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:05.819509    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8804-01fe52"
Oct 30 00:49:06.716826 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:49:06.707931    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod233a53ca_0f95_49b1_86e0_bebf6591475b.slice/crio-99c1363082edf02c754ae40e01547f0127923029f5e5c7bd8161ebfaf08fe03b WatchSource:0}: Error finding container 99c1363082edf02c754ae40e01547f0127923029f5e5c7bd8161ebfaf08fe03b: Status 404 returned error can't find the container with id 99c1363082edf02c754ae40e01547f0127923029f5e5c7bd8161ebfaf08fe03b
Oct 30 00:49:06.782994 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:06.760468    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8804-01fe52"]
Oct 30 00:49:06.813815 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:06.805301    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8804-7c671b" podUID="96bfc194-7371-4338-b143-64e6329e9469" containerName="test-curl-8804-7c671b" containerID="cri-o://de19130ac87b01bfa87177bb224431eccff8c0491a5656681128c1678e2c74ec" gracePeriod=2
Oct 30 00:49:07.402803 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:07.401881    4393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g" podUID="fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:07.716982 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:07.715940    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g" podUID="fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 30 00:49:07.742805 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:07.732934    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q" podUID="46b01388-7da7-4e63-95f3-afc811bed013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:07.870250 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:07.870207    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8804-01fe52" event={"ID":"233a53ca-0f95-49b1-86e0-bebf6591475b","Type":"ContainerStarted","Data":"99c1363082edf02c754ae40e01547f0127923029f5e5c7bd8161ebfaf08fe03b"}
Oct 30 00:49:08.112862 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:08.111870    4393 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x" podUID="28f3e4ff-5875-4e5d-8cab-60d1a5ba97be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:08.630814 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:08.596879    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x" podUID="28f3e4ff-5875-4e5d-8cab-60d1a5ba97be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:08.742803 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:08.738801    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q" podUID="46b01388-7da7-4e63-95f3-afc811bed013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:08.742803 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:08.739606    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q"
Oct 30 00:49:08.742803 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:08.741273    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g" podUID="fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:08.742803 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:08.741674    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"
Oct 30 00:49:09.615326 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:09.615274    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x" podUID="28f3e4ff-5875-4e5d-8cab-60d1a5ba97be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:09.622120 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:09.621510    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"
Oct 30 00:49:09.630803 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:09.628691    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"
Oct 30 00:49:09.751980 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:09.751933    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q" podUID="46b01388-7da7-4e63-95f3-afc811bed013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:09.961042 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:09.961012    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"
Oct 30 00:49:09.961291 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:09.961280    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q"
Oct 30 00:49:10.394192 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:10.394008    4393 generic.go:356] "Generic (PLEG): container finished" podID="96bfc194-7371-4338-b143-64e6329e9469" containerID="de19130ac87b01bfa87177bb224431eccff8c0491a5656681128c1678e2c74ec" exitCode=137
Oct 30 00:49:10.397381 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:10.397346    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8804-01fe52" event={"ID":"233a53ca-0f95-49b1-86e0-bebf6591475b","Type":"ContainerStarted","Data":"fdc3cdc5156a170e02540b1fd2084e34fc596c604ad9ec71758fdb6aa0d3d853"}
Oct 30 00:49:10.437849 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:10.437703    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8804-01fe52" podStartSLOduration=5.43768398 podStartE2EDuration="5.43768398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:10.433887557 +0000 UTC m=+1040.821563948" watchObservedRunningTime="2026-10-30 00:49:10.43768398 +0000 UTC m=+1040.825360369"
Oct 30 00:49:10.538927 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:10.538903    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8804-7c671b"
Oct 30 00:49:10.552064 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:49:10.552035    4393 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8804-7c671b\" is forbidden: User \"system:node:ip-10-0-135-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-56.ec2.internal' and this object" podUID="96bfc194-7371-4338-b143-64e6329e9469" pod="opendatahub/test-curl-8804-7c671b"
Oct 30 00:49:10.642560 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:10.642530    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96bfc194-7371-4338-b143-64e6329e9469-kube-api-access-5vtch\" (UniqueName: \"kubernetes.io/projected/96bfc194-7371-4338-b143-64e6329e9469-kube-api-access-5vtch\") pod \"96bfc194-7371-4338-b143-64e6329e9469\" (UID: \"96bfc194-7371-4338-b143-64e6329e9469\") "
Oct 30 00:49:10.645344 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:10.645311    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96bfc194-7371-4338-b143-64e6329e9469-kube-api-access-5vtch" pod "96bfc194-7371-4338-b143-64e6329e9469" (UID: "96bfc194-7371-4338-b143-64e6329e9469"). InnerVolumeSpecName "kube-api-access-5vtch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:49:10.745218 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:10.745187    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vtch\" (UniqueName: \"kubernetes.io/projected/96bfc194-7371-4338-b143-64e6329e9469-kube-api-access-5vtch\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:49:11.397984 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:11.397937    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8804-01fe52"]
Oct 30 00:49:11.405661 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:11.405627    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8804-01fe52"]
Oct 30 00:49:11.409277 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:11.408770    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8804-7c671b"
Oct 30 00:49:11.409277 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:11.408903    4393 scope.go:122] "RemoveContainer" containerID="de19130ac87b01bfa87177bb224431eccff8c0491a5656681128c1678e2c74ec"
Oct 30 00:49:11.421959 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:49:11.421930    4393 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8804-7c671b\" is forbidden: User \"system:node:ip-10-0-135-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-56.ec2.internal' and this object" podUID="96bfc194-7371-4338-b143-64e6329e9469" pod="opendatahub/test-curl-8804-7c671b"
Oct 30 00:49:11.434900 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:49:11.434854    4393 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8804-7c671b\" is forbidden: User \"system:node:ip-10-0-135-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-56.ec2.internal' and this object" podUID="96bfc194-7371-4338-b143-64e6329e9469" pod="opendatahub/test-curl-8804-7c671b"
Oct 30 00:49:11.726265 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:11.726172    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96bfc194-7371-4338-b143-64e6329e9469" path="/var/lib/kubelet/pods/96bfc194-7371-4338-b143-64e6329e9469/volumes"
Oct 30 00:49:12.417075 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:12.417039    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8804-01fe52" podUID="233a53ca-0f95-49b1-86e0-bebf6591475b" containerName="test-curl-8804-01fe52" containerID="cri-o://fdc3cdc5156a170e02540b1fd2084e34fc596c604ad9ec71758fdb6aa0d3d853" gracePeriod=2
Oct 30 00:49:14.640960 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:14.640938    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8804-01fe52"
Oct 30 00:49:14.740794 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:14.740701    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/233a53ca-0f95-49b1-86e0-bebf6591475b-kube-api-access-c96kx\" (UniqueName: \"kubernetes.io/projected/233a53ca-0f95-49b1-86e0-bebf6591475b-kube-api-access-c96kx\") pod \"233a53ca-0f95-49b1-86e0-bebf6591475b\" (UID: \"233a53ca-0f95-49b1-86e0-bebf6591475b\") "
Oct 30 00:49:14.743490 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:14.743449    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/233a53ca-0f95-49b1-86e0-bebf6591475b-kube-api-access-c96kx" pod "233a53ca-0f95-49b1-86e0-bebf6591475b" (UID: "233a53ca-0f95-49b1-86e0-bebf6591475b"). InnerVolumeSpecName "kube-api-access-c96kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:49:14.842713 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:14.842680    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c96kx\" (UniqueName: \"kubernetes.io/projected/233a53ca-0f95-49b1-86e0-bebf6591475b-kube-api-access-c96kx\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:49:15.117676 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:15.117592    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj"]
Oct 30 00:49:15.118709 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:15.118679    4393 cpu_manager.go:414] "containerMap: removing container" podUID="233a53ca-0f95-49b1-86e0-bebf6591475b" containerName="test-curl-8804-01fe52"
Oct 30 00:49:15.118709 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:15.118709    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="233a53ca-0f95-49b1-86e0-bebf6591475b" containerName="test-curl-8804-01fe52"
Oct 30 00:49:15.118996 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:15.118975    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="233a53ca-0f95-49b1-86e0-bebf6591475b" containerName="test-curl-8804-01fe52"
Oct 30 00:49:15.129858 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:15.123374    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj"
Oct 30 00:49:15.146262 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:15.146236    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj"]
Oct 30 00:49:15.151669 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:15.151639    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-29be73\"/\"kube-root-ca.crt\""
Oct 30 00:49:15.152957 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:15.152934    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-29be73\"/\"openshift-service-ca.crt\""
Oct 30 00:49:15.154945 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:15.154925    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-29be73\"/\"default-dockercfg-vwhx2\""
Oct 30 00:49:15.155046 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:15.154951    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-29be73\"/\"distinct-w1-29be73-kserve-self-signed-certs\""
Oct 30 00:49:15.254561 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:15.254510    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fc7af8a-93f1-4792-b170-90c675c8a919-model-cache\") pod \"distinct-w1-29be73-kserve-867cdb7458-ncqxj\" (UID: \"2fc7af8a-93f1-4792-b170-90c675c8a919\") " pod="e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj"
Oct 30 00:49:15.254745 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:15.254585    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6pp2\" (UniqueName: \"kubernetes.io/projected/2fc7af8a-93f1-4792-b170-90c675c8a919-kube-api-access-r6pp2\") pod \"distinct-w1-29be73-kserve-867cdb7458-ncqxj\" (UID: \"2fc7af8a-93f1-4792-b170-90c675c8a919\") " pod="e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj"
Oct 30 00:49:15.254745 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:15.254733    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fc7af8a-93f1-4792-b170-90c675c8a919-tmp-dir\") pod \"distinct-w1-29be73-kserve-867cdb7458-ncqxj\" (UID: \"2fc7af8a-93f1-4792-b170-90c675c8a919\") " pod="e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj"
Oct 30 00:49:15.254893 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:15.254858    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fc7af8a-93f1-4792-b170-90c675c8a919-dshm\") pod \"distinct-w1-29be73-kserve-867cdb7458-ncqxj\" (UID: \"2fc7af8a-93f1-4792-b170-90c675c8a919\") " pod="e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj"
Oct 30 00:49:15.254942 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:15.254911    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fc7af8a-93f1-4792-b170-90c675c8a919-tls-certs\") pod \"distinct-w1-29be73-kserve-867cdb7458-ncqxj\" (UID: \"2fc7af8a-93f1-4792-b170-90c675c8a919\") " pod="e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj"
Oct 30 00:49:15.254999 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:15.254967    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fc7af8a-93f1-4792-b170-90c675c8a919-home\") pod \"distinct-w1-29be73-kserve-867cdb7458-ncqxj\" (UID: \"2fc7af8a-93f1-4792-b170-90c675c8a919\") " pod="e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj"
Oct 30 00:49:15.378860 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:15.359923    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fc7af8a-93f1-4792-b170-90c675c8a919-home\") pod \"distinct-w1-29be73-kserve-867cdb7458-ncqxj\" (UID: \"2fc7af8a-93f1-4792-b170-90c675c8a919\") " pod="e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj"
Oct 30 00:49:15.378860 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:15.360056    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fc7af8a-93f1-4792-b170-90c675c8a919-model-cache\") pod \"distinct-w1-29be73-kserve-867cdb7458-ncqxj\" (UID: \"2fc7af8a-93f1-4792-b170-90c675c8a919\") " pod="e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj"
Oct 30 00:49:15.378860 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:15.360109    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6pp2\" (UniqueName: \"kubernetes.io/projected/2fc7af8a-93f1-4792-b170-90c675c8a919-kube-api-access-r6pp2\") pod \"distinct-w1-29be73-kserve-867cdb7458-ncqxj\" (UID: \"2fc7af8a-93f1-4792-b170-90c675c8a919\") " pod="e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj"
Oct 30 00:49:15.378860 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:15.360194    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fc7af8a-93f1-4792-b170-90c675c8a919-tmp-dir\") pod \"distinct-w1-29be73-kserve-867cdb7458-ncqxj\" (UID: \"2fc7af8a-93f1-4792-b170-90c675c8a919\") " pod="e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj"
Oct 30 00:49:15.378860 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:15.360309    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fc7af8a-93f1-4792-b170-90c675c8a919-dshm\") pod \"distinct-w1-29be73-kserve-867cdb7458-ncqxj\" (UID: \"2fc7af8a-93f1-4792-b170-90c675c8a919\") " pod="e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj"
Oct 30 00:49:15.378860 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:15.360363    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fc7af8a-93f1-4792-b170-90c675c8a919-tls-certs\") pod \"distinct-w1-29be73-kserve-867cdb7458-ncqxj\" (UID: \"2fc7af8a-93f1-4792-b170-90c675c8a919\") " pod="e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj"
Oct 30 00:49:15.411800 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:15.405707    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fc7af8a-93f1-4792-b170-90c675c8a919-home\") pod \"distinct-w1-29be73-kserve-867cdb7458-ncqxj\" (UID: \"2fc7af8a-93f1-4792-b170-90c675c8a919\") " pod="e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj"
Oct 30 00:49:15.412472 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:15.412438    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fc7af8a-93f1-4792-b170-90c675c8a919-model-cache\") pod \"distinct-w1-29be73-kserve-867cdb7458-ncqxj\" (UID: \"2fc7af8a-93f1-4792-b170-90c675c8a919\") " pod="e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj"
Oct 30 00:49:15.431591 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:15.413018    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fc7af8a-93f1-4792-b170-90c675c8a919-tmp-dir\") pod \"distinct-w1-29be73-kserve-867cdb7458-ncqxj\" (UID: \"2fc7af8a-93f1-4792-b170-90c675c8a919\") " pod="e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj"
Oct 30 00:49:15.505805 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:15.503276    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fc7af8a-93f1-4792-b170-90c675c8a919-tls-certs\") pod \"distinct-w1-29be73-kserve-867cdb7458-ncqxj\" (UID: \"2fc7af8a-93f1-4792-b170-90c675c8a919\") " pod="e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj"
Oct 30 00:49:15.505805 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:15.503421    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fc7af8a-93f1-4792-b170-90c675c8a919-dshm\") pod \"distinct-w1-29be73-kserve-867cdb7458-ncqxj\" (UID: \"2fc7af8a-93f1-4792-b170-90c675c8a919\") " pod="e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj"
Oct 30 00:49:15.515868 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:15.506640    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6pp2\" (UniqueName: \"kubernetes.io/projected/2fc7af8a-93f1-4792-b170-90c675c8a919-kube-api-access-r6pp2\") pod \"distinct-w1-29be73-kserve-867cdb7458-ncqxj\" (UID: \"2fc7af8a-93f1-4792-b170-90c675c8a919\") " pod="e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj"
Oct 30 00:49:15.551747 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:15.551710    4393 generic.go:356] "Generic (PLEG): container finished" podID="233a53ca-0f95-49b1-86e0-bebf6591475b" containerID="fdc3cdc5156a170e02540b1fd2084e34fc596c604ad9ec71758fdb6aa0d3d853" exitCode=137
Oct 30 00:49:15.557820 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:15.552146    4393 scope.go:122] "RemoveContainer" containerID="fdc3cdc5156a170e02540b1fd2084e34fc596c604ad9ec71758fdb6aa0d3d853"
Oct 30 00:49:15.558211 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:15.558185    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8804-01fe52"
Oct 30 00:49:15.627268 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:15.627138    4393 scope.go:122] "RemoveContainer" containerID="fdc3cdc5156a170e02540b1fd2084e34fc596c604ad9ec71758fdb6aa0d3d853"
Oct 30 00:49:15.629948 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:49:15.629830    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdc3cdc5156a170e02540b1fd2084e34fc596c604ad9ec71758fdb6aa0d3d853\": container with ID starting with fdc3cdc5156a170e02540b1fd2084e34fc596c604ad9ec71758fdb6aa0d3d853 not found: ID does not exist" containerID="fdc3cdc5156a170e02540b1fd2084e34fc596c604ad9ec71758fdb6aa0d3d853"
Oct 30 00:49:15.629948 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:15.629871    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdc3cdc5156a170e02540b1fd2084e34fc596c604ad9ec71758fdb6aa0d3d853"} err="failed to get container status \"fdc3cdc5156a170e02540b1fd2084e34fc596c604ad9ec71758fdb6aa0d3d853\": rpc error: code = NotFound desc = could not find container \"fdc3cdc5156a170e02540b1fd2084e34fc596c604ad9ec71758fdb6aa0d3d853\": container with ID starting with fdc3cdc5156a170e02540b1fd2084e34fc596c604ad9ec71758fdb6aa0d3d853 not found: ID does not exist"
Oct 30 00:49:15.663185 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:49:15.663076    4393 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8804-01fe52\" is forbidden: User \"system:node:ip-10-0-135-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-56.ec2.internal' and this object" podUID="233a53ca-0f95-49b1-86e0-bebf6591475b" pod="opendatahub/test-curl-8804-01fe52"
Oct 30 00:49:15.728044 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:15.728002    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="233a53ca-0f95-49b1-86e0-bebf6591475b" path="/var/lib/kubelet/pods/233a53ca-0f95-49b1-86e0-bebf6591475b/volumes"
Oct 30 00:49:15.744262 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:15.744234    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj"
Oct 30 00:49:16.153033 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:16.152992    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj"]
Oct 30 00:49:16.562517 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:16.562421    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj" event={"ID":"2fc7af8a-93f1-4792-b170-90c675c8a919","Type":"ContainerStarted","Data":"3a20f1761ac7c5c9e24f6cc42fa65d4f5042d40568b3ebdc47e6df7f5bfd0bc9"}
Oct 30 00:49:16.562517 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:16.562469    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj" event={"ID":"2fc7af8a-93f1-4792-b170-90c675c8a919","Type":"ContainerStarted","Data":"89b9277ce05c03cc39beb0541ae61c491e5f60502d705fd82e00520785164e82"}
Oct 30 00:49:16.606109 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:16.606060    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj" podStartSLOduration=1.606046385 podStartE2EDuration="1.606046385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:16.602469503 +0000 UTC m=+1046.990145892" watchObservedRunningTime="2026-10-30 00:49:16.606046385 +0000 UTC m=+1046.993722773"
Oct 30 00:49:16.755834 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:16.753770    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj"
Oct 30 00:49:17.863197 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:17.863164    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3ba85458-646f477df9-4xctc"]
Oct 30 00:49:17.868065 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:17.868043    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ba85458-646f477df9-4xctc"
Oct 30 00:49:17.897294 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:17.897255    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttgdl\" (UniqueName: \"kubernetes.io/projected/966264d7-a7c5-41c0-9c7e-dafd22a3d2ab-kube-api-access-ttgdl\") pod \"payload-pre-processing-e2e-ait-3ba85458-646f477df9-4xctc\" (UID: \"966264d7-a7c5-41c0-9c7e-dafd22a3d2ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3ba85458-646f477df9-4xctc"
Oct 30 00:49:17.897560 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:17.897406    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/966264d7-a7c5-41c0-9c7e-dafd22a3d2ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3ba85458-646f477df9-4xctc\" (UID: \"966264d7-a7c5-41c0-9c7e-dafd22a3d2ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3ba85458-646f477df9-4xctc"
Oct 30 00:49:17.906698 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:17.906662    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3ba85458\""
Oct 30 00:49:17.906698 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:17.906665    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3ba85458-dockercfg-nflqb\""
Oct 30 00:49:17.911838 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:17.911813    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3ba85458-646f477df9-4xctc"]
Oct 30 00:49:17.939355 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:17.939330    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3ba85458-7cf86cd866-6xz69"]
Oct 30 00:49:17.953060 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:17.953035    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3ba85458-7cf86cd866-6xz69"
Oct 30 00:49:17.965339 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:17.965312    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3ba85458-7cf86cd866-6xz69"]
Oct 30 00:49:17.999105 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:17.999069    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttgdl\" (UniqueName: \"kubernetes.io/projected/966264d7-a7c5-41c0-9c7e-dafd22a3d2ab-kube-api-access-ttgdl\") pod \"payload-pre-processing-e2e-ait-3ba85458-646f477df9-4xctc\" (UID: \"966264d7-a7c5-41c0-9c7e-dafd22a3d2ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3ba85458-646f477df9-4xctc"
Oct 30 00:49:17.999105 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:17.999108    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/125e2185-b66d-4853-94b5-1961ed27fc63-plugins-config-volume\") pod \"payload-processing-e2e-ait-3ba85458-7cf86cd866-6xz69\" (UID: \"125e2185-b66d-4853-94b5-1961ed27fc63\") " pod="openshift-ingress/payload-processing-e2e-ait-3ba85458-7cf86cd866-6xz69"
Oct 30 00:49:17.999355 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:17.999149    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84kh2\" (UniqueName: \"kubernetes.io/projected/125e2185-b66d-4853-94b5-1961ed27fc63-kube-api-access-84kh2\") pod \"payload-processing-e2e-ait-3ba85458-7cf86cd866-6xz69\" (UID: \"125e2185-b66d-4853-94b5-1961ed27fc63\") " pod="openshift-ingress/payload-processing-e2e-ait-3ba85458-7cf86cd866-6xz69"
Oct 30 00:49:17.999355 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:17.999239    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/966264d7-a7c5-41c0-9c7e-dafd22a3d2ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3ba85458-646f477df9-4xctc\" (UID: \"966264d7-a7c5-41c0-9c7e-dafd22a3d2ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3ba85458-646f477df9-4xctc"
Oct 30 00:49:17.999963 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:17.999936    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/966264d7-a7c5-41c0-9c7e-dafd22a3d2ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3ba85458-646f477df9-4xctc\" (UID: \"966264d7-a7c5-41c0-9c7e-dafd22a3d2ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3ba85458-646f477df9-4xctc"
Oct 30 00:49:18.090806 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:18.080533    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttgdl\" (UniqueName: \"kubernetes.io/projected/966264d7-a7c5-41c0-9c7e-dafd22a3d2ab-kube-api-access-ttgdl\") pod \"payload-pre-processing-e2e-ait-3ba85458-646f477df9-4xctc\" (UID: \"966264d7-a7c5-41c0-9c7e-dafd22a3d2ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3ba85458-646f477df9-4xctc"
Oct 30 00:49:18.120860 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:18.115055    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/125e2185-b66d-4853-94b5-1961ed27fc63-plugins-config-volume\") pod \"payload-processing-e2e-ait-3ba85458-7cf86cd866-6xz69\" (UID: \"125e2185-b66d-4853-94b5-1961ed27fc63\") " pod="openshift-ingress/payload-processing-e2e-ait-3ba85458-7cf86cd866-6xz69"
Oct 30 00:49:18.120860 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:18.115122    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84kh2\" (UniqueName: \"kubernetes.io/projected/125e2185-b66d-4853-94b5-1961ed27fc63-kube-api-access-84kh2\") pod \"payload-processing-e2e-ait-3ba85458-7cf86cd866-6xz69\" (UID: \"125e2185-b66d-4853-94b5-1961ed27fc63\") " pod="openshift-ingress/payload-processing-e2e-ait-3ba85458-7cf86cd866-6xz69"
Oct 30 00:49:18.120860 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:18.116847    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/125e2185-b66d-4853-94b5-1961ed27fc63-plugins-config-volume\") pod \"payload-processing-e2e-ait-3ba85458-7cf86cd866-6xz69\" (UID: \"125e2185-b66d-4853-94b5-1961ed27fc63\") " pod="openshift-ingress/payload-processing-e2e-ait-3ba85458-7cf86cd866-6xz69"
Oct 30 00:49:18.163808 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:18.158580    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84kh2\" (UniqueName: \"kubernetes.io/projected/125e2185-b66d-4853-94b5-1961ed27fc63-kube-api-access-84kh2\") pod \"payload-processing-e2e-ait-3ba85458-7cf86cd866-6xz69\" (UID: \"125e2185-b66d-4853-94b5-1961ed27fc63\") " pod="openshift-ingress/payload-processing-e2e-ait-3ba85458-7cf86cd866-6xz69"
Oct 30 00:49:18.194873 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:18.191142    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ba85458-646f477df9-4xctc"
Oct 30 00:49:18.305487 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:18.304235    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3ba85458-7cf86cd866-6xz69"
Oct 30 00:49:18.611043 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:18.611008    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3ba85458-646f477df9-4xctc"]
Oct 30 00:49:18.617337 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:49:18.616692    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod966264d7_a7c5_41c0_9c7e_dafd22a3d2ab.slice/crio-e7c405593273250a99ee64d57dfb2d2002df55e377b588ffb4a11a5c2c1e9955 WatchSource:0}: Error finding container e7c405593273250a99ee64d57dfb2d2002df55e377b588ffb4a11a5c2c1e9955: Status 404 returned error can't find the container with id e7c405593273250a99ee64d57dfb2d2002df55e377b588ffb4a11a5c2c1e9955
Oct 30 00:49:18.638913 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:18.638883    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3ba85458-7cf86cd866-6xz69"]
Oct 30 00:49:19.589920 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:19.589883    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3ba85458-7cf86cd866-6xz69" event={"ID":"125e2185-b66d-4853-94b5-1961ed27fc63","Type":"ContainerStarted","Data":"ef055d3febec9fe2cdcc430f2da254e8f39188879b69a06a491b8aacb4a3f66e"}
Oct 30 00:49:19.590438 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:19.589948    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3ba85458-7cf86cd866-6xz69" event={"ID":"125e2185-b66d-4853-94b5-1961ed27fc63","Type":"ContainerStarted","Data":"8e70c81a10772e1a99ab7e9e08c8f2120d0c94842ded3ea0cab055f83b0ae122"}
Oct 30 00:49:19.590438 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:19.589981    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-3ba85458-7cf86cd866-6xz69"
Oct 30 00:49:19.591812 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:19.591769    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ba85458-646f477df9-4xctc" event={"ID":"966264d7-a7c5-41c0-9c7e-dafd22a3d2ab","Type":"ContainerStarted","Data":"5e1ad42b561a50f9ad9f761820a70e1dad512b828cf1fa2a780651444ca306cc"}
Oct 30 00:49:19.591933 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:19.591844    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ba85458-646f477df9-4xctc" event={"ID":"966264d7-a7c5-41c0-9c7e-dafd22a3d2ab","Type":"ContainerStarted","Data":"e7c405593273250a99ee64d57dfb2d2002df55e377b588ffb4a11a5c2c1e9955"}
Oct 30 00:49:19.591933 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:19.591877    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ba85458-646f477df9-4xctc"
Oct 30 00:49:19.634809 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:19.634711    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-3ba85458-7cf86cd866-6xz69" podStartSLOduration=2.6346930200000003 podStartE2EDuration="2.63469302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:19.633022163 +0000 UTC m=+1050.020698545" watchObservedRunningTime="2026-10-30 00:49:19.63469302 +0000 UTC m=+1050.022369410"
Oct 30 00:49:19.675648 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:19.675595    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ba85458-646f477df9-4xctc" podStartSLOduration=2.675581155 podStartE2EDuration="2.675581155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:19.672456887 +0000 UTC m=+1050.060133278" watchObservedRunningTime="2026-10-30 00:49:19.675581155 +0000 UTC m=+1050.063257841"
Oct 30 00:49:20.563804 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:20.563754    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cab9990d-76b4b665cf-75mnn"]
Oct 30 00:49:20.569014 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:20.568992    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cab9990d-76b4b665cf-75mnn"
Oct 30 00:49:20.587048 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:20.587020    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cab9990d-76b4b665cf-75mnn"]
Oct 30 00:49:20.593559 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:20.593533    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-cab9990d-dockercfg-62jdq\""
Oct 30 00:49:20.593559 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:20.593533    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-cab9990d\""
Oct 30 00:49:20.722814 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:20.718600    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrdgk\" (UniqueName: \"kubernetes.io/projected/a802164a-cc5f-4c52-9047-e69c63002b66-kube-api-access-jrdgk\") pod \"payload-pre-processing-e2e-migrate-cab9990d-76b4b665cf-75mnn\" (UID: \"a802164a-cc5f-4c52-9047-e69c63002b66\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cab9990d-76b4b665cf-75mnn"
Oct 30 00:49:20.722814 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:20.718705    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a802164a-cc5f-4c52-9047-e69c63002b66-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-cab9990d-76b4b665cf-75mnn\" (UID: \"a802164a-cc5f-4c52-9047-e69c63002b66\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cab9990d-76b4b665cf-75mnn"
Oct 30 00:49:20.885479 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:20.823426    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a802164a-cc5f-4c52-9047-e69c63002b66-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-cab9990d-76b4b665cf-75mnn\" (UID: \"a802164a-cc5f-4c52-9047-e69c63002b66\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cab9990d-76b4b665cf-75mnn"
Oct 30 00:49:20.885479 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:20.823891    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrdgk\" (UniqueName: \"kubernetes.io/projected/a802164a-cc5f-4c52-9047-e69c63002b66-kube-api-access-jrdgk\") pod \"payload-pre-processing-e2e-migrate-cab9990d-76b4b665cf-75mnn\" (UID: \"a802164a-cc5f-4c52-9047-e69c63002b66\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cab9990d-76b4b665cf-75mnn"
Oct 30 00:49:20.885479 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:20.825206    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a802164a-cc5f-4c52-9047-e69c63002b66-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-cab9990d-76b4b665cf-75mnn\" (UID: \"a802164a-cc5f-4c52-9047-e69c63002b66\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cab9990d-76b4b665cf-75mnn"
Oct 30 00:49:20.918808 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:20.918723    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cab9990d-75bddb75b6-nsjnn"]
Oct 30 00:49:20.942928 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:20.928470    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cab9990d-75bddb75b6-nsjnn"
Oct 30 00:49:20.942928 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:20.935611    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrdgk\" (UniqueName: \"kubernetes.io/projected/a802164a-cc5f-4c52-9047-e69c63002b66-kube-api-access-jrdgk\") pod \"payload-pre-processing-e2e-migrate-cab9990d-76b4b665cf-75mnn\" (UID: \"a802164a-cc5f-4c52-9047-e69c63002b66\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cab9990d-76b4b665cf-75mnn"
Oct 30 00:49:20.953507 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:20.951599    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cab9990d-75bddb75b6-nsjnn"]
Oct 30 00:49:21.029823 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:21.029767    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09c71ec7-db60-4886-9e2b-878a1bbf659b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-cab9990d-75bddb75b6-nsjnn\" (UID: \"09c71ec7-db60-4886-9e2b-878a1bbf659b\") " pod="openshift-ingress/payload-processing-e2e-migrate-cab9990d-75bddb75b6-nsjnn"
Oct 30 00:49:21.030000 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:21.029910    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxrlw\" (UniqueName: \"kubernetes.io/projected/09c71ec7-db60-4886-9e2b-878a1bbf659b-kube-api-access-cxrlw\") pod \"payload-processing-e2e-migrate-cab9990d-75bddb75b6-nsjnn\" (UID: \"09c71ec7-db60-4886-9e2b-878a1bbf659b\") " pod="openshift-ingress/payload-processing-e2e-migrate-cab9990d-75bddb75b6-nsjnn"
Oct 30 00:49:21.131810 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:21.131741    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09c71ec7-db60-4886-9e2b-878a1bbf659b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-cab9990d-75bddb75b6-nsjnn\" (UID: \"09c71ec7-db60-4886-9e2b-878a1bbf659b\") " pod="openshift-ingress/payload-processing-e2e-migrate-cab9990d-75bddb75b6-nsjnn"
Oct 30 00:49:21.131983 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:21.131829    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxrlw\" (UniqueName: \"kubernetes.io/projected/09c71ec7-db60-4886-9e2b-878a1bbf659b-kube-api-access-cxrlw\") pod \"payload-processing-e2e-migrate-cab9990d-75bddb75b6-nsjnn\" (UID: \"09c71ec7-db60-4886-9e2b-878a1bbf659b\") " pod="openshift-ingress/payload-processing-e2e-migrate-cab9990d-75bddb75b6-nsjnn"
Oct 30 00:49:21.132470 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:21.132449    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09c71ec7-db60-4886-9e2b-878a1bbf659b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-cab9990d-75bddb75b6-nsjnn\" (UID: \"09c71ec7-db60-4886-9e2b-878a1bbf659b\") " pod="openshift-ingress/payload-processing-e2e-migrate-cab9990d-75bddb75b6-nsjnn"
Oct 30 00:49:21.154875 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:21.154805    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxrlw\" (UniqueName: \"kubernetes.io/projected/09c71ec7-db60-4886-9e2b-878a1bbf659b-kube-api-access-cxrlw\") pod \"payload-processing-e2e-migrate-cab9990d-75bddb75b6-nsjnn\" (UID: \"09c71ec7-db60-4886-9e2b-878a1bbf659b\") " pod="openshift-ingress/payload-processing-e2e-migrate-cab9990d-75bddb75b6-nsjnn"
Oct 30 00:49:21.185909 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:21.185872    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cab9990d-76b4b665cf-75mnn"
Oct 30 00:49:21.269822 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:21.267101    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cab9990d-75bddb75b6-nsjnn"
Oct 30 00:49:21.498263 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:49:21.498234    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda802164a_cc5f_4c52_9047_e69c63002b66.slice/crio-79466099823ffa1232edacb5e68783ce29aab3b3cede0a6fa01ee5e89e20fdeb WatchSource:0}: Error finding container 79466099823ffa1232edacb5e68783ce29aab3b3cede0a6fa01ee5e89e20fdeb: Status 404 returned error can't find the container with id 79466099823ffa1232edacb5e68783ce29aab3b3cede0a6fa01ee5e89e20fdeb
Oct 30 00:49:21.498464 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:21.498444    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cab9990d-76b4b665cf-75mnn"]
Oct 30 00:49:21.662877 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:49:21.660235    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c71ec7_db60_4886_9e2b_878a1bbf659b.slice/crio-09d48723f534770495d15d2b6b410a90fb38a27eaf02e47c575b91b720a663c8 WatchSource:0}: Error finding container 09d48723f534770495d15d2b6b410a90fb38a27eaf02e47c575b91b720a663c8: Status 404 returned error can't find the container with id 09d48723f534770495d15d2b6b410a90fb38a27eaf02e47c575b91b720a663c8
Oct 30 00:49:21.677093 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:21.676339    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cab9990d-75bddb75b6-nsjnn"]
Oct 30 00:49:21.678823 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:21.678076    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cab9990d-76b4b665cf-75mnn" event={"ID":"a802164a-cc5f-4c52-9047-e69c63002b66","Type":"ContainerStarted","Data":"79466099823ffa1232edacb5e68783ce29aab3b3cede0a6fa01ee5e89e20fdeb"}
Oct 30 00:49:22.687190 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:22.687153    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cab9990d-76b4b665cf-75mnn" event={"ID":"a802164a-cc5f-4c52-9047-e69c63002b66","Type":"ContainerStarted","Data":"6346739217c1ebff606995a64b1f0e36fd01da9174637210e1b174ee4f43f84f"}
Oct 30 00:49:22.687624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:22.687352    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cab9990d-76b4b665cf-75mnn"
Oct 30 00:49:22.688978 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:22.688949    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cab9990d-75bddb75b6-nsjnn" event={"ID":"09c71ec7-db60-4886-9e2b-878a1bbf659b","Type":"ContainerStarted","Data":"e1095483026025b25affd4df9e3c36e648aac1168eb9b5eb9af90e19694f709f"}
Oct 30 00:49:22.689105 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:22.688996    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cab9990d-75bddb75b6-nsjnn" event={"ID":"09c71ec7-db60-4886-9e2b-878a1bbf659b","Type":"ContainerStarted","Data":"09d48723f534770495d15d2b6b410a90fb38a27eaf02e47c575b91b720a663c8"}
Oct 30 00:49:22.689105 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:22.689043    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-cab9990d-75bddb75b6-nsjnn"
Oct 30 00:49:22.729425 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:22.729380    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cab9990d-76b4b665cf-75mnn" podStartSLOduration=2.7293490179999997 podStartE2EDuration="2.729349018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:22.726057485 +0000 UTC m=+1053.113733872" watchObservedRunningTime="2026-10-30 00:49:22.729349018 +0000 UTC m=+1053.117025407"
Oct 30 00:49:22.768470 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:22.768427    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-cab9990d-75bddb75b6-nsjnn" podStartSLOduration=2.768413163 podStartE2EDuration="2.768413163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:22.765206675 +0000 UTC m=+1053.152883054" watchObservedRunningTime="2026-10-30 00:49:22.768413163 +0000 UTC m=+1053.156089551"
Oct 30 00:49:25.829807 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:25.801676    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q" podUID="46b01388-7da7-4e63-95f3-afc811bed013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Oct 30 00:49:25.829807 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:25.807859    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g" podUID="fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:25.885435 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:25.885401    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj"
Oct 30 00:49:25.975684 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:25.975645    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x" podUID="28f3e4ff-5875-4e5d-8cab-60d1a5ba97be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Oct 30 00:49:26.778805 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:26.762221    4393 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj" podUID="2fc7af8a-93f1-4792-b170-90c675c8a919" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": context deadline exceeded"
Oct 30 00:49:26.843803 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:26.812859    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q" podUID="46b01388-7da7-4e63-95f3-afc811bed013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:26.843803 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:26.813335    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g" podUID="fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:26.843803 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:26.833419    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q"
Oct 30 00:49:26.843803 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:26.833487    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"
Oct 30 00:49:26.976986 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:26.976935    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x" podUID="28f3e4ff-5875-4e5d-8cab-60d1a5ba97be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:26.982928 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:26.982894    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"
Oct 30 00:49:27.059095 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:27.058958    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q"
Oct 30 00:49:27.081177 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:27.081148    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"
Oct 30 00:49:27.094135 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:49:27.094100    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:49:27.102562 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:27.101946    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"
Oct 30 00:49:30.605422 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:30.605392    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ba85458-646f477df9-4xctc"
Oct 30 00:49:30.605826 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:30.605474    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-3ba85458-7cf86cd866-6xz69"
Oct 30 00:49:31.519206 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:31.519164    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8"]
Oct 30 00:49:31.527844 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:31.527796    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8"
Oct 30 00:49:31.541243 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:31.541162    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8"]
Oct 30 00:49:31.548615 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:31.548589    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3ba85458-serving-cert\""
Oct 30 00:49:31.645611 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:31.645572    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cjt6\" (UniqueName: \"kubernetes.io/projected/b71b75d8-eb01-4748-a17f-2cc7b69f8614-kube-api-access-4cjt6\") pod \"maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8\" (UID: \"b71b75d8-eb01-4748-a17f-2cc7b69f8614\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8"
Oct 30 00:49:31.646140 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:31.645830    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b71b75d8-eb01-4748-a17f-2cc7b69f8614-maas-api-tls\") pod \"maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8\" (UID: \"b71b75d8-eb01-4748-a17f-2cc7b69f8614\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8"
Oct 30 00:49:31.646140 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:31.646026    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b71b75d8-eb01-4748-a17f-2cc7b69f8614-openshift-service-ca\") pod \"maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8\" (UID: \"b71b75d8-eb01-4748-a17f-2cc7b69f8614\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8"
Oct 30 00:49:31.646140 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:31.646060    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b71b75d8-eb01-4748-a17f-2cc7b69f8614-metrics-certs\") pod \"maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8\" (UID: \"b71b75d8-eb01-4748-a17f-2cc7b69f8614\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8"
Oct 30 00:49:31.750802 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:31.750093    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b71b75d8-eb01-4748-a17f-2cc7b69f8614-openshift-service-ca\") pod \"maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8\" (UID: \"b71b75d8-eb01-4748-a17f-2cc7b69f8614\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8"
Oct 30 00:49:31.750802 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:31.750146    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b71b75d8-eb01-4748-a17f-2cc7b69f8614-metrics-certs\") pod \"maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8\" (UID: \"b71b75d8-eb01-4748-a17f-2cc7b69f8614\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8"
Oct 30 00:49:31.750802 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:31.750252    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cjt6\" (UniqueName: \"kubernetes.io/projected/b71b75d8-eb01-4748-a17f-2cc7b69f8614-kube-api-access-4cjt6\") pod \"maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8\" (UID: \"b71b75d8-eb01-4748-a17f-2cc7b69f8614\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8"
Oct 30 00:49:31.750802 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:31.750393    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b71b75d8-eb01-4748-a17f-2cc7b69f8614-maas-api-tls\") pod \"maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8\" (UID: \"b71b75d8-eb01-4748-a17f-2cc7b69f8614\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8"
Oct 30 00:49:31.766595 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:31.756247    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b71b75d8-eb01-4748-a17f-2cc7b69f8614-openshift-service-ca\") pod \"maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8\" (UID: \"b71b75d8-eb01-4748-a17f-2cc7b69f8614\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8"
Oct 30 00:49:31.766595 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:31.762424    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b71b75d8-eb01-4748-a17f-2cc7b69f8614-metrics-certs\") pod \"maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8\" (UID: \"b71b75d8-eb01-4748-a17f-2cc7b69f8614\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8"
Oct 30 00:49:31.785847 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:31.785168    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b71b75d8-eb01-4748-a17f-2cc7b69f8614-maas-api-tls\") pod \"maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8\" (UID: \"b71b75d8-eb01-4748-a17f-2cc7b69f8614\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8"
Oct 30 00:49:31.786154 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:31.786109    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cjt6\" (UniqueName: \"kubernetes.io/projected/b71b75d8-eb01-4748-a17f-2cc7b69f8614-kube-api-access-4cjt6\") pod \"maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8\" (UID: \"b71b75d8-eb01-4748-a17f-2cc7b69f8614\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8"
Oct 30 00:49:31.853416 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:31.853266    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8"
Oct 30 00:49:32.165702 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:49:32.165677    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb71b75d8_eb01_4748_a17f_2cc7b69f8614.slice/crio-43dd5e64e3bc4c955297b1d155a19257b1491e1651d9b9b582a84ff275c38555 WatchSource:0}: Error finding container 43dd5e64e3bc4c955297b1d155a19257b1491e1651d9b9b582a84ff275c38555: Status 404 returned error can't find the container with id 43dd5e64e3bc4c955297b1d155a19257b1491e1651d9b9b582a84ff275c38555
Oct 30 00:49:32.165879 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:32.165853    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8"]
Oct 30 00:49:33.179896 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:33.179858    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8" event={"ID":"b71b75d8-eb01-4748-a17f-2cc7b69f8614","Type":"ContainerStarted","Data":"915fbf94b996b72e4b5b04d69796e88702311e1b93441a918294f065b7859627"}
Oct 30 00:49:33.180294 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:33.179919    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8" event={"ID":"b71b75d8-eb01-4748-a17f-2cc7b69f8614","Type":"ContainerStarted","Data":"43dd5e64e3bc4c955297b1d155a19257b1491e1651d9b9b582a84ff275c38555"}
Oct 30 00:49:33.180294 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:33.180015    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8"
Oct 30 00:49:33.222071 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:33.222016    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8" podStartSLOduration=1.878103518 podStartE2EDuration="2.222001675s" totalImagesPullingTime="343.898157ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:33.220163576 +0000 UTC m=+1063.607839966" watchObservedRunningTime="2026-10-30 00:49:33.222001675 +0000 UTC m=+1063.609678065"
Oct 30 00:49:33.698294 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:33.698262    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cab9990d-76b4b665cf-75mnn"
Oct 30 00:49:33.698479 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:33.698399    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-cab9990d-75bddb75b6-nsjnn"
Oct 30 00:49:35.458835 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:35.458794    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"]
Oct 30 00:49:35.463543 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:35.463520    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"
Oct 30 00:49:35.485015 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:35.484986    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"]
Oct 30 00:49:35.491712 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:35.491685    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-cefb27-openshift-default-dockercfg-jwt4c\""
Oct 30 00:49:35.560484 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:35.560454    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whrx6\" (UniqueName: \"kubernetes.io/projected/39473464-58b6-4589-a88d-eaa60276d7f0-kube-api-access-whrx6\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5\" (UID: \"39473464-58b6-4589-a88d-eaa60276d7f0\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"
Oct 30 00:49:35.560640 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:35.560521    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39473464-58b6-4589-a88d-eaa60276d7f0-istio-podinfo\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5\" (UID: \"39473464-58b6-4589-a88d-eaa60276d7f0\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"
Oct 30 00:49:35.560640 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:35.560613    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39473464-58b6-4589-a88d-eaa60276d7f0-credential-socket\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5\" (UID: \"39473464-58b6-4589-a88d-eaa60276d7f0\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"
Oct 30 00:49:35.560723 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:35.560667    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39473464-58b6-4589-a88d-eaa60276d7f0-workload-certs\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5\" (UID: \"39473464-58b6-4589-a88d-eaa60276d7f0\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"
Oct 30 00:49:35.560820 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:35.560805    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39473464-58b6-4589-a88d-eaa60276d7f0-istio-data\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5\" (UID: \"39473464-58b6-4589-a88d-eaa60276d7f0\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"
Oct 30 00:49:35.560954 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:35.560931    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39473464-58b6-4589-a88d-eaa60276d7f0-istio-token\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5\" (UID: \"39473464-58b6-4589-a88d-eaa60276d7f0\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"
Oct 30 00:49:35.560990 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:35.560975    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39473464-58b6-4589-a88d-eaa60276d7f0-workload-socket\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5\" (UID: \"39473464-58b6-4589-a88d-eaa60276d7f0\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"
Oct 30 00:49:35.561101 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:35.561087    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39473464-58b6-4589-a88d-eaa60276d7f0-istiod-ca-cert\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5\" (UID: \"39473464-58b6-4589-a88d-eaa60276d7f0\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"
Oct 30 00:49:35.561161 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:35.561147    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39473464-58b6-4589-a88d-eaa60276d7f0-istio-envoy\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5\" (UID: \"39473464-58b6-4589-a88d-eaa60276d7f0\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"
Oct 30 00:49:35.662677 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:35.662639    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39473464-58b6-4589-a88d-eaa60276d7f0-workload-certs\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5\" (UID: \"39473464-58b6-4589-a88d-eaa60276d7f0\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"
Oct 30 00:49:35.662873 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:35.662722    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39473464-58b6-4589-a88d-eaa60276d7f0-istio-data\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5\" (UID: \"39473464-58b6-4589-a88d-eaa60276d7f0\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"
Oct 30 00:49:35.662873 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:35.662840    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39473464-58b6-4589-a88d-eaa60276d7f0-istio-token\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5\" (UID: \"39473464-58b6-4589-a88d-eaa60276d7f0\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"
Oct 30 00:49:35.662873 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:35.662865    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39473464-58b6-4589-a88d-eaa60276d7f0-workload-socket\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5\" (UID: \"39473464-58b6-4589-a88d-eaa60276d7f0\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"
Oct 30 00:49:35.662980 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:35.662926    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39473464-58b6-4589-a88d-eaa60276d7f0-istiod-ca-cert\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5\" (UID: \"39473464-58b6-4589-a88d-eaa60276d7f0\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"
Oct 30 00:49:35.663017 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:35.662984    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39473464-58b6-4589-a88d-eaa60276d7f0-istio-envoy\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5\" (UID: \"39473464-58b6-4589-a88d-eaa60276d7f0\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"
Oct 30 00:49:35.663057 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:35.663019    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39473464-58b6-4589-a88d-eaa60276d7f0-workload-certs\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5\" (UID: \"39473464-58b6-4589-a88d-eaa60276d7f0\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"
Oct 30 00:49:35.663118 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:35.663095    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39473464-58b6-4589-a88d-eaa60276d7f0-istio-data\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5\" (UID: \"39473464-58b6-4589-a88d-eaa60276d7f0\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"
Oct 30 00:49:35.663169 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:35.663100    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whrx6\" (UniqueName: \"kubernetes.io/projected/39473464-58b6-4589-a88d-eaa60276d7f0-kube-api-access-whrx6\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5\" (UID: \"39473464-58b6-4589-a88d-eaa60276d7f0\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"
Oct 30 00:49:35.663206 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:35.663190    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39473464-58b6-4589-a88d-eaa60276d7f0-istio-podinfo\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5\" (UID: \"39473464-58b6-4589-a88d-eaa60276d7f0\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"
Oct 30 00:49:35.663245 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:35.663222    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39473464-58b6-4589-a88d-eaa60276d7f0-workload-socket\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5\" (UID: \"39473464-58b6-4589-a88d-eaa60276d7f0\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"
Oct 30 00:49:35.663301 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:35.663241    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39473464-58b6-4589-a88d-eaa60276d7f0-credential-socket\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5\" (UID: \"39473464-58b6-4589-a88d-eaa60276d7f0\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"
Oct 30 00:49:35.663539 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:35.663518    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39473464-58b6-4589-a88d-eaa60276d7f0-credential-socket\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5\" (UID: \"39473464-58b6-4589-a88d-eaa60276d7f0\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"
Oct 30 00:49:35.663637 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:35.663600    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39473464-58b6-4589-a88d-eaa60276d7f0-istiod-ca-cert\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5\" (UID: \"39473464-58b6-4589-a88d-eaa60276d7f0\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"
Oct 30 00:49:35.665247 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:35.665231    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39473464-58b6-4589-a88d-eaa60276d7f0-istio-envoy\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5\" (UID: \"39473464-58b6-4589-a88d-eaa60276d7f0\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"
Oct 30 00:49:35.665682 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:35.665665    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39473464-58b6-4589-a88d-eaa60276d7f0-istio-podinfo\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5\" (UID: \"39473464-58b6-4589-a88d-eaa60276d7f0\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"
Oct 30 00:49:35.686686 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:35.686660    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whrx6\" (UniqueName: \"kubernetes.io/projected/39473464-58b6-4589-a88d-eaa60276d7f0-kube-api-access-whrx6\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5\" (UID: \"39473464-58b6-4589-a88d-eaa60276d7f0\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"
Oct 30 00:49:35.686822 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:35.686753    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39473464-58b6-4589-a88d-eaa60276d7f0-istio-token\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5\" (UID: \"39473464-58b6-4589-a88d-eaa60276d7f0\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"
Oct 30 00:49:35.757491 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:35.757406    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj"
Oct 30 00:49:35.769452 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:35.769428    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj"
Oct 30 00:49:35.779474 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:35.779451    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"
Oct 30 00:49:36.144189 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:36.144150    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"]
Oct 30 00:49:36.144316 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:49:36.144147    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39473464_58b6_4589_a88d_eaa60276d7f0.slice/crio-441da9bf7d5f8cd9f9b1eaabcc8c1a24a913014b0015634736345f7640048599 WatchSource:0}: Error finding container 441da9bf7d5f8cd9f9b1eaabcc8c1a24a913014b0015634736345f7640048599: Status 404 returned error can't find the container with id 441da9bf7d5f8cd9f9b1eaabcc8c1a24a913014b0015634736345f7640048599
Oct 30 00:49:36.147031 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:36.146996    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:49:36.147135 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:36.147071    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:49:36.147135 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:36.147114    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:49:36.265678 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:36.265637    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5" event={"ID":"39473464-58b6-4589-a88d-eaa60276d7f0","Type":"ContainerStarted","Data":"453a0f7d160fee1cbe7ccd561a2e8fb46670273e83eb7185355944b4fd71a91e"}
Oct 30 00:49:36.265887 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:36.265705    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5" event={"ID":"39473464-58b6-4589-a88d-eaa60276d7f0","Type":"ContainerStarted","Data":"441da9bf7d5f8cd9f9b1eaabcc8c1a24a913014b0015634736345f7640048599"}
Oct 30 00:49:36.312387 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:36.312333    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5" podStartSLOduration=1.31231863 podStartE2EDuration="1.31231863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:36.30960223 +0000 UTC m=+1066.697278618" watchObservedRunningTime="2026-10-30 00:49:36.31231863 +0000 UTC m=+1066.699995019"
Oct 30 00:49:36.529396 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:36.529301    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj"]
Oct 30 00:49:36.534138 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:36.534114    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj"
Oct 30 00:49:36.553999 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:36.553971    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-cab9990d-serving-cert\""
Oct 30 00:49:36.558092 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:36.557898    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj"]
Oct 30 00:49:36.582800 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:36.582632    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3176c8ca-243b-4ad6-8b53-845e9a8843a6-openshift-service-ca\") pod \"maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj\" (UID: \"3176c8ca-243b-4ad6-8b53-845e9a8843a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj"
Oct 30 00:49:36.582800 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:36.582721    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3176c8ca-243b-4ad6-8b53-845e9a8843a6-maas-api-tls\") pod \"maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj\" (UID: \"3176c8ca-243b-4ad6-8b53-845e9a8843a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj"
Oct 30 00:49:36.582800 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:36.582750    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6zj6\" (UniqueName: \"kubernetes.io/projected/3176c8ca-243b-4ad6-8b53-845e9a8843a6-kube-api-access-x6zj6\") pod \"maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj\" (UID: \"3176c8ca-243b-4ad6-8b53-845e9a8843a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj"
Oct 30 00:49:36.583094 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:36.582881    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3176c8ca-243b-4ad6-8b53-845e9a8843a6-metrics-certs\") pod \"maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj\" (UID: \"3176c8ca-243b-4ad6-8b53-845e9a8843a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj"
Oct 30 00:49:36.684161 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:36.684131    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3176c8ca-243b-4ad6-8b53-845e9a8843a6-openshift-service-ca\") pod \"maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj\" (UID: \"3176c8ca-243b-4ad6-8b53-845e9a8843a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj"
Oct 30 00:49:36.684330 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:36.684224    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3176c8ca-243b-4ad6-8b53-845e9a8843a6-maas-api-tls\") pod \"maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj\" (UID: \"3176c8ca-243b-4ad6-8b53-845e9a8843a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj"
Oct 30 00:49:36.684330 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:36.684244    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6zj6\" (UniqueName: \"kubernetes.io/projected/3176c8ca-243b-4ad6-8b53-845e9a8843a6-kube-api-access-x6zj6\") pod \"maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj\" (UID: \"3176c8ca-243b-4ad6-8b53-845e9a8843a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj"
Oct 30 00:49:36.684330 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:36.684313    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3176c8ca-243b-4ad6-8b53-845e9a8843a6-metrics-certs\") pod \"maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj\" (UID: \"3176c8ca-243b-4ad6-8b53-845e9a8843a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj"
Oct 30 00:49:36.684843 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:36.684821    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3176c8ca-243b-4ad6-8b53-845e9a8843a6-openshift-service-ca\") pod \"maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj\" (UID: \"3176c8ca-243b-4ad6-8b53-845e9a8843a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj"
Oct 30 00:49:36.686919 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:36.686895    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3176c8ca-243b-4ad6-8b53-845e9a8843a6-maas-api-tls\") pod \"maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj\" (UID: \"3176c8ca-243b-4ad6-8b53-845e9a8843a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj"
Oct 30 00:49:36.687049 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:36.687031    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3176c8ca-243b-4ad6-8b53-845e9a8843a6-metrics-certs\") pod \"maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj\" (UID: \"3176c8ca-243b-4ad6-8b53-845e9a8843a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj"
Oct 30 00:49:36.708599 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:36.708156    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6zj6\" (UniqueName: \"kubernetes.io/projected/3176c8ca-243b-4ad6-8b53-845e9a8843a6-kube-api-access-x6zj6\") pod \"maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj\" (UID: \"3176c8ca-243b-4ad6-8b53-845e9a8843a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj"
Oct 30 00:49:36.784277 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:36.779934    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"
Oct 30 00:49:36.880809 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:36.880341    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj"
Oct 30 00:49:37.257172 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:49:37.257144    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3176c8ca_243b_4ad6_8b53_845e9a8843a6.slice/crio-edf73212dd05f3d89209a62f6ad4cd49d577688e8f2732dabb170b9778172d18 WatchSource:0}: Error finding container edf73212dd05f3d89209a62f6ad4cd49d577688e8f2732dabb170b9778172d18: Status 404 returned error can't find the container with id edf73212dd05f3d89209a62f6ad4cd49d577688e8f2732dabb170b9778172d18
Oct 30 00:49:37.257737 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:37.257713    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj"]
Oct 30 00:49:37.273521 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:37.273490    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj" event={"ID":"3176c8ca-243b-4ad6-8b53-845e9a8843a6","Type":"ContainerStarted","Data":"edf73212dd05f3d89209a62f6ad4cd49d577688e8f2732dabb170b9778172d18"}
Oct 30 00:49:37.794806 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:37.792984    4393 patch_prober.go:29] interesting pod/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 00:49:37.794806 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:37.793045    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5" podUID="39473464-58b6-4589-a88d-eaa60276d7f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 00:49:38.282424 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:38.282387    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj" event={"ID":"3176c8ca-243b-4ad6-8b53-845e9a8843a6","Type":"ContainerStarted","Data":"cba7af3bdf0616ad7970c58f1e1b427d3c0bcdc9d69b2579f33cf58d403aacbd"}
Oct 30 00:49:38.351835 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:38.351764    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj" podStartSLOduration=2.019363328 podStartE2EDuration="2.351750582s" totalImagesPullingTime="332.387254ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:38.348638259 +0000 UTC m=+1068.736314664" watchObservedRunningTime="2026-10-30 00:49:38.351750582 +0000 UTC m=+1068.739427022"
Oct 30 00:49:38.780048 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:38.780011    4393 patch_prober.go:29] interesting pod/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 00:49:38.780335 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:38.780074    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5" podUID="39473464-58b6-4589-a88d-eaa60276d7f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 00:49:39.193129 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:39.193101    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8"
Oct 30 00:49:39.294996 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:39.294967    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj"
Oct 30 00:49:39.780829 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:39.780770    4393 patch_prober.go:29] interesting pod/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 00:49:39.781037 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:39.780842    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5" podUID="39473464-58b6-4589-a88d-eaa60276d7f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 00:49:40.779981 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:40.779948    4393 patch_prober.go:29] interesting pod/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 00:49:40.780415 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:40.779998    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5" podUID="39473464-58b6-4589-a88d-eaa60276d7f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 00:49:41.779839 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:41.779808    4393 patch_prober.go:29] interesting pod/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 00:49:41.780059 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:41.779862    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5" podUID="39473464-58b6-4589-a88d-eaa60276d7f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 00:49:42.780435 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:42.780405    4393 patch_prober.go:29] interesting pod/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 00:49:42.780858 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:42.780463    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5" podUID="39473464-58b6-4589-a88d-eaa60276d7f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 00:49:43.785534 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:43.785500    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"
Oct 30 00:49:43.786000 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:43.785977    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"
Oct 30 00:49:43.786754 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:43.786734    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"
Oct 30 00:49:45.321876 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:45.321841    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj"
Oct 30 00:49:50.741519 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:50.741481    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"]
Oct 30 00:49:50.842276 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:50.842239    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"]
Oct 30 00:49:50.842609 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:50.842590    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"
Oct 30 00:49:50.993868 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:50.943046    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914501a6-32b3-4158-9171-8320ce9291d4-istiod-ca-cert\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx\" (UID: \"914501a6-32b3-4158-9171-8320ce9291d4\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"
Oct 30 00:49:51.031752 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:51.031720    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914501a6-32b3-4158-9171-8320ce9291d4-workload-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx\" (UID: \"914501a6-32b3-4158-9171-8320ce9291d4\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"
Oct 30 00:49:51.031972 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:51.031949    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914501a6-32b3-4158-9171-8320ce9291d4-credential-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx\" (UID: \"914501a6-32b3-4158-9171-8320ce9291d4\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"
Oct 30 00:49:51.057181 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:51.057130    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914501a6-32b3-4158-9171-8320ce9291d4-istio-data\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx\" (UID: \"914501a6-32b3-4158-9171-8320ce9291d4\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"
Oct 30 00:49:51.057649 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:51.057619    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914501a6-32b3-4158-9171-8320ce9291d4-workload-certs\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx\" (UID: \"914501a6-32b3-4158-9171-8320ce9291d4\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"
Oct 30 00:49:51.140016 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:51.127100    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914501a6-32b3-4158-9171-8320ce9291d4-istio-podinfo\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx\" (UID: \"914501a6-32b3-4158-9171-8320ce9291d4\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"
Oct 30 00:49:51.140016 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:51.127191    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gnvl\" (UniqueName: \"kubernetes.io/projected/914501a6-32b3-4158-9171-8320ce9291d4-kube-api-access-6gnvl\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx\" (UID: \"914501a6-32b3-4158-9171-8320ce9291d4\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"
Oct 30 00:49:51.140016 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:51.127231    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914501a6-32b3-4158-9171-8320ce9291d4-istio-envoy\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx\" (UID: \"914501a6-32b3-4158-9171-8320ce9291d4\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"
Oct 30 00:49:51.140016 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:51.127256    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914501a6-32b3-4158-9171-8320ce9291d4-istio-token\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx\" (UID: \"914501a6-32b3-4158-9171-8320ce9291d4\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"
Oct 30 00:49:51.274485 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:51.274389    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gnvl\" (UniqueName: \"kubernetes.io/projected/914501a6-32b3-4158-9171-8320ce9291d4-kube-api-access-6gnvl\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx\" (UID: \"914501a6-32b3-4158-9171-8320ce9291d4\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"
Oct 30 00:49:51.318144 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:51.318110    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914501a6-32b3-4158-9171-8320ce9291d4-istio-envoy\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx\" (UID: \"914501a6-32b3-4158-9171-8320ce9291d4\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"
Oct 30 00:49:51.318402 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:51.318381    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914501a6-32b3-4158-9171-8320ce9291d4-istio-token\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx\" (UID: \"914501a6-32b3-4158-9171-8320ce9291d4\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"
Oct 30 00:49:51.318574 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:51.318557    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914501a6-32b3-4158-9171-8320ce9291d4-istiod-ca-cert\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx\" (UID: \"914501a6-32b3-4158-9171-8320ce9291d4\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"
Oct 30 00:49:51.340026 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:51.339993    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914501a6-32b3-4158-9171-8320ce9291d4-workload-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx\" (UID: \"914501a6-32b3-4158-9171-8320ce9291d4\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"
Oct 30 00:49:51.340276 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:51.340257    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914501a6-32b3-4158-9171-8320ce9291d4-credential-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx\" (UID: \"914501a6-32b3-4158-9171-8320ce9291d4\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"
Oct 30 00:49:51.340420 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:51.340403    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914501a6-32b3-4158-9171-8320ce9291d4-istio-data\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx\" (UID: \"914501a6-32b3-4158-9171-8320ce9291d4\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"
Oct 30 00:49:51.373670 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:51.340739    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914501a6-32b3-4158-9171-8320ce9291d4-workload-certs\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx\" (UID: \"914501a6-32b3-4158-9171-8320ce9291d4\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"
Oct 30 00:49:51.392212 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:51.392177    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914501a6-32b3-4158-9171-8320ce9291d4-istio-podinfo\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx\" (UID: \"914501a6-32b3-4158-9171-8320ce9291d4\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"
Oct 30 00:49:51.392528 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:51.331092    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914501a6-32b3-4158-9171-8320ce9291d4-istiod-ca-cert\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx\" (UID: \"914501a6-32b3-4158-9171-8320ce9291d4\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"
Oct 30 00:49:51.393310 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:51.372409    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914501a6-32b3-4158-9171-8320ce9291d4-istio-data\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx\" (UID: \"914501a6-32b3-4158-9171-8320ce9291d4\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"
Oct 30 00:49:51.393475 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:51.372145    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914501a6-32b3-4158-9171-8320ce9291d4-workload-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx\" (UID: \"914501a6-32b3-4158-9171-8320ce9291d4\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"
Oct 30 00:49:51.393606 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:51.393103    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914501a6-32b3-4158-9171-8320ce9291d4-workload-certs\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx\" (UID: \"914501a6-32b3-4158-9171-8320ce9291d4\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"
Oct 30 00:49:51.398424 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:51.398366    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914501a6-32b3-4158-9171-8320ce9291d4-credential-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx\" (UID: \"914501a6-32b3-4158-9171-8320ce9291d4\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"
Oct 30 00:49:51.445558 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:51.425886    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gnvl\" (UniqueName: \"kubernetes.io/projected/914501a6-32b3-4158-9171-8320ce9291d4-kube-api-access-6gnvl\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx\" (UID: \"914501a6-32b3-4158-9171-8320ce9291d4\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"
Oct 30 00:49:51.446113 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:51.435347    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914501a6-32b3-4158-9171-8320ce9291d4-istio-token\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx\" (UID: \"914501a6-32b3-4158-9171-8320ce9291d4\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"
Oct 30 00:49:51.526802 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:51.525330    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914501a6-32b3-4158-9171-8320ce9291d4-istio-podinfo\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx\" (UID: \"914501a6-32b3-4158-9171-8320ce9291d4\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"
Oct 30 00:49:51.526802 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:51.525561    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914501a6-32b3-4158-9171-8320ce9291d4-istio-envoy\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx\" (UID: \"914501a6-32b3-4158-9171-8320ce9291d4\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"
Oct 30 00:49:51.624568 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:51.624311    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"
Oct 30 00:49:52.839860 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:52.800102    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj" podUID="2fc7af8a-93f1-4792-b170-90c675c8a919" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:53.065955 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:53.065904    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q" podUID="46b01388-7da7-4e63-95f3-afc811bed013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Oct 30 00:49:53.090984 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:53.090881    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g" podUID="fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Oct 30 00:49:53.105048 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:53.105002    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x" podUID="28f3e4ff-5875-4e5d-8cab-60d1a5ba97be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:49:53.346911 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:53.335034    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"]
Oct 30 00:49:53.425649 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:49:53.425603    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod914501a6_32b3_4158_9171_8320ce9291d4.slice/crio-0878f976cd8531e2040c332d2d313cf9f7dee0a5ddecb28d43daa3b2da5fca48 WatchSource:0}: Error finding container 0878f976cd8531e2040c332d2d313cf9f7dee0a5ddecb28d43daa3b2da5fca48: Status 404 returned error can't find the container with id 0878f976cd8531e2040c332d2d313cf9f7dee0a5ddecb28d43daa3b2da5fca48
Oct 30 00:49:53.546009 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:53.506141    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:49:53.546009 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:53.506243    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:49:53.546009 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:53.506291    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:49:54.128080 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:54.128041    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx" event={"ID":"914501a6-32b3-4158-9171-8320ce9291d4","Type":"ContainerStarted","Data":"4b3389fa5dc65f2f7f7e14ea8d839ab90e80abfba04a50a11cb7b2d0b8c01bc0"}
Oct 30 00:49:54.128692 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:54.128657    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx" event={"ID":"914501a6-32b3-4158-9171-8320ce9291d4","Type":"ContainerStarted","Data":"0878f976cd8531e2040c332d2d313cf9f7dee0a5ddecb28d43daa3b2da5fca48"}
Oct 30 00:49:54.187329 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:54.187266    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx" podStartSLOduration=4.187246619 podStartE2EDuration="4.187246619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:49:54.184209553 +0000 UTC m=+1084.571885942" watchObservedRunningTime="2026-10-30 00:49:54.187246619 +0000 UTC m=+1084.574923009"
Oct 30 00:49:54.634290 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:54.634250    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"
Oct 30 00:49:54.635473 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:54.635449    4393 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 00:49:54.635593 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:54.635494    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx" podUID="914501a6-32b3-4158-9171-8320ce9291d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 00:49:55.633991 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:55.633952    4393 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 00:49:55.634419 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:55.634019    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx" podUID="914501a6-32b3-4158-9171-8320ce9291d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 00:49:56.547995 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:49:56.547955    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:49:56.634059 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:56.634018    4393 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 00:49:56.634548 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:56.634092    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx" podUID="914501a6-32b3-4158-9171-8320ce9291d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 00:49:57.634723 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:57.634685    4393 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 00:49:57.635299 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:57.634745    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx" podUID="914501a6-32b3-4158-9171-8320ce9291d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 00:49:58.634669 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:58.634633    4393 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 00:49:58.634903 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:58.634688    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx" podUID="914501a6-32b3-4158-9171-8320ce9291d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 00:49:59.648805 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:59.648460    4393 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 00:49:59.648805 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:49:59.648536    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx" podUID="914501a6-32b3-4158-9171-8320ce9291d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 00:50:00.434176 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:00.433618    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"]
Oct 30 00:50:00.441655 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:00.441620    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"
Oct 30 00:50:00.453350 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:00.453312    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"]
Oct 30 00:50:00.555265 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:00.555232    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-cab9990d-gw-openshift-default-dockercfg-7h89s\""
Oct 30 00:50:00.597118 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:00.597077    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-workload-certs\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944\" (UID: \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"
Oct 30 00:50:00.597318 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:00.597156    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgzxt\" (UniqueName: \"kubernetes.io/projected/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-kube-api-access-jgzxt\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944\" (UID: \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"
Oct 30 00:50:00.597318 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:00.597183    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-istio-podinfo\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944\" (UID: \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"
Oct 30 00:50:00.597318 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:00.597220    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-credential-socket\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944\" (UID: \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"
Oct 30 00:50:00.597318 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:00.597302    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-istio-token\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944\" (UID: \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"
Oct 30 00:50:00.597505 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:00.597347    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-istio-envoy\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944\" (UID: \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"
Oct 30 00:50:00.597505 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:00.597386    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-workload-socket\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944\" (UID: \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"
Oct 30 00:50:00.597505 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:00.597413    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-istio-data\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944\" (UID: \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"
Oct 30 00:50:00.597650 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:00.597618    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-istiod-ca-cert\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944\" (UID: \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"
Oct 30 00:50:00.633730 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:00.633689    4393 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 00:50:00.633944 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:00.633744    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx" podUID="914501a6-32b3-4158-9171-8320ce9291d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 00:50:00.698666 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:00.698562    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-workload-certs\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944\" (UID: \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"
Oct 30 00:50:00.699202 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:00.698679    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgzxt\" (UniqueName: \"kubernetes.io/projected/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-kube-api-access-jgzxt\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944\" (UID: \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"
Oct 30 00:50:00.699202 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:00.698711    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-istio-podinfo\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944\" (UID: \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"
Oct 30 00:50:00.699202 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:00.698758    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-credential-socket\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944\" (UID: \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"
Oct 30 00:50:00.699202 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:00.698818    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-istio-token\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944\" (UID: \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"
Oct 30 00:50:00.699202 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:00.698958    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-istio-envoy\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944\" (UID: \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"
Oct 30 00:50:00.699202 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:00.699044    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-workload-socket\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944\" (UID: \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"
Oct 30 00:50:00.699202 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:00.699090    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-istio-data\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944\" (UID: \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"
Oct 30 00:50:00.699202 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:00.699126    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-workload-certs\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944\" (UID: \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"
Oct 30 00:50:00.699607 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:00.699217    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-credential-socket\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944\" (UID: \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"
Oct 30 00:50:00.699607 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:00.699329    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-istiod-ca-cert\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944\" (UID: \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"
Oct 30 00:50:00.699607 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:00.699356    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-workload-socket\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944\" (UID: \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"
Oct 30 00:50:00.699725 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:00.699623    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-istio-data\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944\" (UID: \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"
Oct 30 00:50:00.700145 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:00.700122    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-istiod-ca-cert\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944\" (UID: \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"
Oct 30 00:50:00.701524 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:00.701500    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-istio-podinfo\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944\" (UID: \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"
Oct 30 00:50:00.701524 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:00.701517    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-istio-envoy\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944\" (UID: \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"
Oct 30 00:50:00.724693 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:00.724661    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-istio-token\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944\" (UID: \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"
Oct 30 00:50:00.724866 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:00.724843    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgzxt\" (UniqueName: \"kubernetes.io/projected/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-kube-api-access-jgzxt\") pod \"e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944\" (UID: \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\") " pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"
Oct 30 00:50:00.774983 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:00.774950    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"
Oct 30 00:50:01.104402 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:50:01.104367    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b2c8be_8f15_4c5b_a9ee_22ac611c1e8a.slice/crio-6d1f5c29e6cb0836a9fe7b08a80f6d00ff966f680a823492167d776c73cc078b WatchSource:0}: Error finding container 6d1f5c29e6cb0836a9fe7b08a80f6d00ff966f680a823492167d776c73cc078b: Status 404 returned error can't find the container with id 6d1f5c29e6cb0836a9fe7b08a80f6d00ff966f680a823492167d776c73cc078b
Oct 30 00:50:01.105899 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:01.105869    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"]
Oct 30 00:50:01.107317 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:01.107284    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:50:01.107431 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:01.107350    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:50:01.107431 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:01.107388    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:50:01.197268 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:01.197231    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944" event={"ID":"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a","Type":"ContainerStarted","Data":"6d1f5c29e6cb0836a9fe7b08a80f6d00ff966f680a823492167d776c73cc078b"}
Oct 30 00:50:01.724807 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:01.724539    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"
Oct 30 00:50:01.730801 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:01.725930    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"
Oct 30 00:50:01.757865 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:01.757750    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"
Oct 30 00:50:02.229907 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:02.229860    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944" event={"ID":"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a","Type":"ContainerStarted","Data":"345b06fbc7d39f68db26f3734592cf0cf08074cdc9b5c6a7678d3a5010b32040"}
Oct 30 00:50:02.276421 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:02.276360    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944" podStartSLOduration=2.276343575 podStartE2EDuration="2.276343575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:02.27256647 +0000 UTC m=+1092.660242859" watchObservedRunningTime="2026-10-30 00:50:02.276343575 +0000 UTC m=+1092.664019964"
Oct 30 00:50:02.775476 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:02.775444    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"
Oct 30 00:50:02.777001 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:02.776977    4393 patch_prober.go:29] interesting pod/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 00:50:02.777133 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:02.777028    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944" podUID="93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 00:50:03.775643 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:03.775611    4393 patch_prober.go:29] interesting pod/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 00:50:03.776101 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:03.775690    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944" podUID="93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 00:50:04.775622 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:04.775588    4393 patch_prober.go:29] interesting pod/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 00:50:04.775816 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:04.775659    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944" podUID="93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 00:50:05.775239 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:05.775204    4393 patch_prober.go:29] interesting pod/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 00:50:05.775416 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:05.775265    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944" podUID="93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 00:50:05.855298 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:05.855265    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"]
Oct 30 00:50:05.861368 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:05.861338    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"
Oct 30 00:50:05.884805 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:05.884761    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"]
Oct 30 00:50:06.034902 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:06.034808    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-istio-token\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x\" (UID: \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"
Oct 30 00:50:06.034902 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:06.034875    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-workload-certs\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x\" (UID: \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"
Oct 30 00:50:06.035125 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:06.034932    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-workload-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x\" (UID: \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"
Oct 30 00:50:06.035125 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:06.035035    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5md4m\" (UniqueName: \"kubernetes.io/projected/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-kube-api-access-5md4m\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x\" (UID: \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"
Oct 30 00:50:06.035125 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:06.035064    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-credential-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x\" (UID: \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"
Oct 30 00:50:06.035285 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:06.035253    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-istio-envoy\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x\" (UID: \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"
Oct 30 00:50:06.035353 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:06.035333    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-istio-data\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x\" (UID: \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"
Oct 30 00:50:06.035411 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:06.035378    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-istiod-ca-cert\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x\" (UID: \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"
Oct 30 00:50:06.035478 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:06.035464    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-istio-podinfo\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x\" (UID: \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"
Oct 30 00:50:06.136791 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:06.136748    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-istio-token\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x\" (UID: \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"
Oct 30 00:50:06.137028 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:06.136878    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-workload-certs\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x\" (UID: \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"
Oct 30 00:50:06.137028 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:06.136922    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-workload-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x\" (UID: \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"
Oct 30 00:50:06.137136 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:06.137051    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5md4m\" (UniqueName: \"kubernetes.io/projected/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-kube-api-access-5md4m\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x\" (UID: \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"
Oct 30 00:50:06.137136 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:06.137103    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-credential-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x\" (UID: \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"
Oct 30 00:50:06.137279 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:06.137255    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-istio-envoy\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x\" (UID: \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"
Oct 30 00:50:06.137442 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:06.137326    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-istio-data\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x\" (UID: \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"
Oct 30 00:50:06.137442 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:06.137371    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-istiod-ca-cert\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x\" (UID: \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"
Oct 30 00:50:06.137570 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:06.137448    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-workload-certs\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x\" (UID: \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"
Oct 30 00:50:06.137570 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:06.137490    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-istio-podinfo\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x\" (UID: \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"
Oct 30 00:50:06.137570 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:06.137519    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-workload-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x\" (UID: \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"
Oct 30 00:50:06.137570 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:06.137554    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-credential-socket\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x\" (UID: \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"
Oct 30 00:50:06.137917 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:06.137870    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-istio-data\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x\" (UID: \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"
Oct 30 00:50:06.138404 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:06.138382    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-istiod-ca-cert\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x\" (UID: \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"
Oct 30 00:50:06.139747 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:06.139727    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-istio-envoy\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x\" (UID: \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"
Oct 30 00:50:06.140181 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:06.140163    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-istio-podinfo\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x\" (UID: \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"
Oct 30 00:50:06.159593 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:06.159569    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5md4m\" (UniqueName: \"kubernetes.io/projected/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-kube-api-access-5md4m\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x\" (UID: \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"
Oct 30 00:50:06.159703 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:06.159673    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-istio-token\") pod \"e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x\" (UID: \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\") " pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"
Oct 30 00:50:06.188696 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:06.188670    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"
Oct 30 00:50:06.541607 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:06.541579    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"]
Oct 30 00:50:06.543183 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:50:06.543157    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc5c9f7_dbe6_4741_8e37_258a1f1ed604.slice/crio-978992d6f119023a29479f1c3012d8c69889361cd709c0f2ada6d53a6755443d WatchSource:0}: Error finding container 978992d6f119023a29479f1c3012d8c69889361cd709c0f2ada6d53a6755443d: Status 404 returned error can't find the container with id 978992d6f119023a29479f1c3012d8c69889361cd709c0f2ada6d53a6755443d
Oct 30 00:50:06.545562 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:06.545529    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:50:06.545648 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:06.545598    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:50:06.545648 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:06.545628    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:50:06.776034 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:06.776004    4393 patch_prober.go:29] interesting pod/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 00:50:06.776194 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:06.776056    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944" podUID="93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 00:50:07.397281 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:07.397239    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x" event={"ID":"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604","Type":"ContainerStarted","Data":"e0ca147f03a24f57bae14caf8fca4797d04d0c40c7e3fbd85fb71f3ea5d8fee0"}
Oct 30 00:50:07.399715 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:07.397311    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x" event={"ID":"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604","Type":"ContainerStarted","Data":"978992d6f119023a29479f1c3012d8c69889361cd709c0f2ada6d53a6755443d"}
Oct 30 00:50:07.492885 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:07.492809    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x" podStartSLOduration=2.492769639 podStartE2EDuration="2.492769639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:07.464303035 +0000 UTC m=+1097.851979437" watchObservedRunningTime="2026-10-30 00:50:07.492769639 +0000 UTC m=+1097.880446029"
Oct 30 00:50:07.494837 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:07.494803    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3ba85458-7cf86cd866-6xz69"]
Oct 30 00:50:07.540048 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:07.540003    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3ba85458-646f477df9-4xctc"]
Oct 30 00:50:07.842867 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:07.822849    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"
Oct 30 00:50:07.842867 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:07.824559    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"
Oct 30 00:50:07.842867 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:07.833079    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"
Oct 30 00:50:08.188848 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:08.188811    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"
Oct 30 00:50:08.189826 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:08.189797    4393 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 00:50:08.189998 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:08.189855    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x" podUID="2bc5c9f7-dbe6-4741-8e37-258a1f1ed604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 00:50:08.837043 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:08.837013    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8"]
Oct 30 00:50:08.837520 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:08.837255    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8" podUID="b71b75d8-eb01-4748-a17f-2cc7b69f8614" containerName="maas-api" containerID="cri-o://915fbf94b996b72e4b5b04d69796e88702311e1b93441a918294f065b7859627" gracePeriod=30
Oct 30 00:50:09.349803 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:09.288936    4393 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 00:50:09.349803 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:09.289002    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x" podUID="2bc5c9f7-dbe6-4741-8e37-258a1f1ed604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 00:50:09.790102 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:09.787028    4393 generic.go:356] "Generic (PLEG): container finished" podID="b71b75d8-eb01-4748-a17f-2cc7b69f8614" containerID="915fbf94b996b72e4b5b04d69796e88702311e1b93441a918294f065b7859627" exitCode=0
Oct 30 00:50:09.837813 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:09.837173    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8"
Oct 30 00:50:09.887680 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:09.887535    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8" event={"ID":"b71b75d8-eb01-4748-a17f-2cc7b69f8614","Type":"ContainerDied","Data":"915fbf94b996b72e4b5b04d69796e88702311e1b93441a918294f065b7859627"}
Oct 30 00:50:09.888339 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:09.888314    4393 scope.go:122] "RemoveContainer" containerID="915fbf94b996b72e4b5b04d69796e88702311e1b93441a918294f065b7859627"
Oct 30 00:50:09.986908 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:09.986875    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b71b75d8-eb01-4748-a17f-2cc7b69f8614-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b71b75d8-eb01-4748-a17f-2cc7b69f8614-openshift-service-ca\") pod \"b71b75d8-eb01-4748-a17f-2cc7b69f8614\" (UID: \"b71b75d8-eb01-4748-a17f-2cc7b69f8614\") "
Oct 30 00:50:09.987102 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:09.986942    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b71b75d8-eb01-4748-a17f-2cc7b69f8614-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b71b75d8-eb01-4748-a17f-2cc7b69f8614-metrics-certs\") pod \"b71b75d8-eb01-4748-a17f-2cc7b69f8614\" (UID: \"b71b75d8-eb01-4748-a17f-2cc7b69f8614\") "
Oct 30 00:50:09.987185 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:09.987115    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b71b75d8-eb01-4748-a17f-2cc7b69f8614-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b71b75d8-eb01-4748-a17f-2cc7b69f8614-maas-api-tls\") pod \"b71b75d8-eb01-4748-a17f-2cc7b69f8614\" (UID: \"b71b75d8-eb01-4748-a17f-2cc7b69f8614\") "
Oct 30 00:50:09.987258 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:09.987240    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b71b75d8-eb01-4748-a17f-2cc7b69f8614-kube-api-access-4cjt6\" (UniqueName: \"kubernetes.io/projected/b71b75d8-eb01-4748-a17f-2cc7b69f8614-kube-api-access-4cjt6\") pod \"b71b75d8-eb01-4748-a17f-2cc7b69f8614\" (UID: \"b71b75d8-eb01-4748-a17f-2cc7b69f8614\") "
Oct 30 00:50:09.987299 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:09.987257    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b71b75d8-eb01-4748-a17f-2cc7b69f8614-openshift-service-ca" pod "b71b75d8-eb01-4748-a17f-2cc7b69f8614" (UID: "b71b75d8-eb01-4748-a17f-2cc7b69f8614"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:50:09.988653 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:09.988627    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b71b75d8-eb01-4748-a17f-2cc7b69f8614-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:50:09.989667 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:09.989637    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b71b75d8-eb01-4748-a17f-2cc7b69f8614-maas-api-tls" pod "b71b75d8-eb01-4748-a17f-2cc7b69f8614" (UID: "b71b75d8-eb01-4748-a17f-2cc7b69f8614"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:50:09.989773 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:09.989667    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b71b75d8-eb01-4748-a17f-2cc7b69f8614-metrics-certs" pod "b71b75d8-eb01-4748-a17f-2cc7b69f8614" (UID: "b71b75d8-eb01-4748-a17f-2cc7b69f8614"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:50:09.989879 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:09.989862    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b71b75d8-eb01-4748-a17f-2cc7b69f8614-kube-api-access-4cjt6" pod "b71b75d8-eb01-4748-a17f-2cc7b69f8614" (UID: "b71b75d8-eb01-4748-a17f-2cc7b69f8614"). InnerVolumeSpecName "kube-api-access-4cjt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:50:10.132806 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:10.125696    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cjt6\" (UniqueName: \"kubernetes.io/projected/b71b75d8-eb01-4748-a17f-2cc7b69f8614-kube-api-access-4cjt6\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:50:10.132806 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:10.125731    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b71b75d8-eb01-4748-a17f-2cc7b69f8614-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:50:10.132806 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:10.125748    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b71b75d8-eb01-4748-a17f-2cc7b69f8614-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:50:10.193515 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:10.191262    4393 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 00:50:10.193515 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:10.191340    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x" podUID="2bc5c9f7-dbe6-4741-8e37-258a1f1ed604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 00:50:10.507864 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:10.502042    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cab9990d-75bddb75b6-nsjnn"]
Oct 30 00:50:10.540417 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:10.540383    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cab9990d-76b4b665cf-75mnn"]
Oct 30 00:50:10.887800 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:10.882920    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8" event={"ID":"b71b75d8-eb01-4748-a17f-2cc7b69f8614","Type":"ContainerDied","Data":"43dd5e64e3bc4c955297b1d155a19257b1491e1651d9b9b582a84ff275c38555"}
Oct 30 00:50:10.887800 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:10.883058    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8"
Oct 30 00:50:10.949026 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:10.948986    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8"]
Oct 30 00:50:10.955884 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:10.955854    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8"]
Oct 30 00:50:11.190053 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:11.189966    4393 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 00:50:11.190053 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:11.190020    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x" podUID="2bc5c9f7-dbe6-4741-8e37-258a1f1ed604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 00:50:11.868806 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:11.860219    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b71b75d8-eb01-4748-a17f-2cc7b69f8614" path="/var/lib/kubelet/pods/b71b75d8-eb01-4748-a17f-2cc7b69f8614/volumes"
Oct 30 00:50:12.189760 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:12.189681    4393 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 00:50:12.189760 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:12.189728    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x" podUID="2bc5c9f7-dbe6-4741-8e37-258a1f1ed604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 00:50:12.345672 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:12.345630    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3ba85458-5bbf4c5776-grss8" podUID="b71b75d8-eb01-4748-a17f-2cc7b69f8614" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.95:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:50:12.496342 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:12.496248    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-3ba85458-7cf86cd866-6xz69" podUID="125e2185-b66d-4853-94b5-1961ed27fc63" containerName="payload-processing" containerID="cri-o://ef055d3febec9fe2cdcc430f2da254e8f39188879b69a06a491b8aacb4a3f66e" gracePeriod=25
Oct 30 00:50:12.540932 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:12.540895    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ba85458-646f477df9-4xctc" podUID="966264d7-a7c5-41c0-9c7e-dafd22a3d2ab" containerName="payload-pre-processing" containerID="cri-o://5e1ad42b561a50f9ad9f761820a70e1dad512b828cf1fa2a780651444ca306cc" gracePeriod=25
Oct 30 00:50:12.947469 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:12.947243    4393 generic.go:356] "Generic (PLEG): container finished" podID="125e2185-b66d-4853-94b5-1961ed27fc63" containerID="ef055d3febec9fe2cdcc430f2da254e8f39188879b69a06a491b8aacb4a3f66e" exitCode=0
Oct 30 00:50:12.947469 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:12.947398    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3ba85458-7cf86cd866-6xz69" event={"ID":"125e2185-b66d-4853-94b5-1961ed27fc63","Type":"ContainerDied","Data":"ef055d3febec9fe2cdcc430f2da254e8f39188879b69a06a491b8aacb4a3f66e"}
Oct 30 00:50:12.949754 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:12.949724    4393 generic.go:356] "Generic (PLEG): container finished" podID="966264d7-a7c5-41c0-9c7e-dafd22a3d2ab" containerID="5e1ad42b561a50f9ad9f761820a70e1dad512b828cf1fa2a780651444ca306cc" exitCode=0
Oct 30 00:50:12.949954 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:12.949934    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ba85458-646f477df9-4xctc" event={"ID":"966264d7-a7c5-41c0-9c7e-dafd22a3d2ab","Type":"ContainerDied","Data":"5e1ad42b561a50f9ad9f761820a70e1dad512b828cf1fa2a780651444ca306cc"}
Oct 30 00:50:13.035810 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:13.035712    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ba85458-646f477df9-4xctc"
Oct 30 00:50:13.039685 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:13.039663    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3ba85458-7cf86cd866-6xz69"
Oct 30 00:50:13.160819 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:13.157765    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/125e2185-b66d-4853-94b5-1961ed27fc63-kube-api-access-84kh2\" (UniqueName: \"kubernetes.io/projected/125e2185-b66d-4853-94b5-1961ed27fc63-kube-api-access-84kh2\") pod \"125e2185-b66d-4853-94b5-1961ed27fc63\" (UID: \"125e2185-b66d-4853-94b5-1961ed27fc63\") "
Oct 30 00:50:13.160819 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:13.157920    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/966264d7-a7c5-41c0-9c7e-dafd22a3d2ab-kube-api-access-ttgdl\" (UniqueName: \"kubernetes.io/projected/966264d7-a7c5-41c0-9c7e-dafd22a3d2ab-kube-api-access-ttgdl\") pod \"966264d7-a7c5-41c0-9c7e-dafd22a3d2ab\" (UID: \"966264d7-a7c5-41c0-9c7e-dafd22a3d2ab\") "
Oct 30 00:50:13.160819 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:13.158043    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/966264d7-a7c5-41c0-9c7e-dafd22a3d2ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/966264d7-a7c5-41c0-9c7e-dafd22a3d2ab-plugins-config-volume\") pod \"966264d7-a7c5-41c0-9c7e-dafd22a3d2ab\" (UID: \"966264d7-a7c5-41c0-9c7e-dafd22a3d2ab\") "
Oct 30 00:50:13.160819 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:13.158147    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/125e2185-b66d-4853-94b5-1961ed27fc63-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/125e2185-b66d-4853-94b5-1961ed27fc63-plugins-config-volume\") pod \"125e2185-b66d-4853-94b5-1961ed27fc63\" (UID: \"125e2185-b66d-4853-94b5-1961ed27fc63\") "
Oct 30 00:50:13.181803 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:13.161445    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/125e2185-b66d-4853-94b5-1961ed27fc63-plugins-config-volume" pod "125e2185-b66d-4853-94b5-1961ed27fc63" (UID: "125e2185-b66d-4853-94b5-1961ed27fc63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:50:13.232808 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:13.215028    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/966264d7-a7c5-41c0-9c7e-dafd22a3d2ab-plugins-config-volume" pod "966264d7-a7c5-41c0-9c7e-dafd22a3d2ab" (UID: "966264d7-a7c5-41c0-9c7e-dafd22a3d2ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:50:13.232808 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:13.215181    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/125e2185-b66d-4853-94b5-1961ed27fc63-kube-api-access-84kh2" pod "125e2185-b66d-4853-94b5-1961ed27fc63" (UID: "125e2185-b66d-4853-94b5-1961ed27fc63"). InnerVolumeSpecName "kube-api-access-84kh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:50:13.232808 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:13.215908    4393 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 00:50:13.232808 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:13.215999    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x" podUID="2bc5c9f7-dbe6-4741-8e37-258a1f1ed604" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 00:50:13.260475 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:13.260390    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/966264d7-a7c5-41c0-9c7e-dafd22a3d2ab-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:50:13.260475 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:13.260423    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/125e2185-b66d-4853-94b5-1961ed27fc63-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:50:13.260475 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:13.260437    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84kh2\" (UniqueName: \"kubernetes.io/projected/125e2185-b66d-4853-94b5-1961ed27fc63-kube-api-access-84kh2\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:50:13.268378 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:13.268329    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/966264d7-a7c5-41c0-9c7e-dafd22a3d2ab-kube-api-access-ttgdl" pod "966264d7-a7c5-41c0-9c7e-dafd22a3d2ab" (UID: "966264d7-a7c5-41c0-9c7e-dafd22a3d2ab"). InnerVolumeSpecName "kube-api-access-ttgdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:50:13.361587 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:13.361493    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttgdl\" (UniqueName: \"kubernetes.io/projected/966264d7-a7c5-41c0-9c7e-dafd22a3d2ab-kube-api-access-ttgdl\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:50:13.959138 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:13.959109    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3ba85458-7cf86cd866-6xz69"
Oct 30 00:50:13.959311 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:13.959165    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3ba85458-7cf86cd866-6xz69" event={"ID":"125e2185-b66d-4853-94b5-1961ed27fc63","Type":"ContainerDied","Data":"8e70c81a10772e1a99ab7e9e08c8f2120d0c94842ded3ea0cab055f83b0ae122"}
Oct 30 00:50:13.959311 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:13.959223    4393 scope.go:122] "RemoveContainer" containerID="ef055d3febec9fe2cdcc430f2da254e8f39188879b69a06a491b8aacb4a3f66e"
Oct 30 00:50:13.961772 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:13.961748    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ba85458-646f477df9-4xctc"
Oct 30 00:50:13.961772 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:13.961754    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ba85458-646f477df9-4xctc" event={"ID":"966264d7-a7c5-41c0-9c7e-dafd22a3d2ab","Type":"ContainerDied","Data":"e7c405593273250a99ee64d57dfb2d2002df55e377b588ffb4a11a5c2c1e9955"}
Oct 30 00:50:13.971886 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:13.971859    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj"]
Oct 30 00:50:13.972108 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:13.972083    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj" podUID="3176c8ca-243b-4ad6-8b53-845e9a8843a6" containerName="maas-api" containerID="cri-o://cba7af3bdf0616ad7970c58f1e1b427d3c0bcdc9d69b2579f33cf58d403aacbd" gracePeriod=30
Oct 30 00:50:13.982486 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:13.978331    4393 scope.go:122] "RemoveContainer" containerID="5e1ad42b561a50f9ad9f761820a70e1dad512b828cf1fa2a780651444ca306cc"
Oct 30 00:50:14.021393 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:14.021359    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3ba85458-7cf86cd866-6xz69"]
Oct 30 00:50:14.029943 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:14.029910    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3ba85458-7cf86cd866-6xz69"]
Oct 30 00:50:14.097238 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:14.097203    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3ba85458-646f477df9-4xctc"]
Oct 30 00:50:14.101531 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:14.101504    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3ba85458-646f477df9-4xctc"]
Oct 30 00:50:14.520759 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:14.520733    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj"
Oct 30 00:50:14.638157 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:14.638119    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3176c8ca-243b-4ad6-8b53-845e9a8843a6-kube-api-access-x6zj6\" (UniqueName: \"kubernetes.io/projected/3176c8ca-243b-4ad6-8b53-845e9a8843a6-kube-api-access-x6zj6\") pod \"3176c8ca-243b-4ad6-8b53-845e9a8843a6\" (UID: \"3176c8ca-243b-4ad6-8b53-845e9a8843a6\") "
Oct 30 00:50:14.638367 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:14.638254    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3176c8ca-243b-4ad6-8b53-845e9a8843a6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3176c8ca-243b-4ad6-8b53-845e9a8843a6-maas-api-tls\") pod \"3176c8ca-243b-4ad6-8b53-845e9a8843a6\" (UID: \"3176c8ca-243b-4ad6-8b53-845e9a8843a6\") "
Oct 30 00:50:14.638367 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:14.638314    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3176c8ca-243b-4ad6-8b53-845e9a8843a6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3176c8ca-243b-4ad6-8b53-845e9a8843a6-metrics-certs\") pod \"3176c8ca-243b-4ad6-8b53-845e9a8843a6\" (UID: \"3176c8ca-243b-4ad6-8b53-845e9a8843a6\") "
Oct 30 00:50:14.638503 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:14.638387    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3176c8ca-243b-4ad6-8b53-845e9a8843a6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3176c8ca-243b-4ad6-8b53-845e9a8843a6-openshift-service-ca\") pod \"3176c8ca-243b-4ad6-8b53-845e9a8843a6\" (UID: \"3176c8ca-243b-4ad6-8b53-845e9a8843a6\") "
Oct 30 00:50:14.639122 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:14.639085    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3176c8ca-243b-4ad6-8b53-845e9a8843a6-openshift-service-ca" pod "3176c8ca-243b-4ad6-8b53-845e9a8843a6" (UID: "3176c8ca-243b-4ad6-8b53-845e9a8843a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:50:14.640693 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:14.640668    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3176c8ca-243b-4ad6-8b53-845e9a8843a6-maas-api-tls" pod "3176c8ca-243b-4ad6-8b53-845e9a8843a6" (UID: "3176c8ca-243b-4ad6-8b53-845e9a8843a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:50:14.641013 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:14.640993    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3176c8ca-243b-4ad6-8b53-845e9a8843a6-metrics-certs" pod "3176c8ca-243b-4ad6-8b53-845e9a8843a6" (UID: "3176c8ca-243b-4ad6-8b53-845e9a8843a6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:50:14.641234 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:14.641215    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3176c8ca-243b-4ad6-8b53-845e9a8843a6-kube-api-access-x6zj6" pod "3176c8ca-243b-4ad6-8b53-845e9a8843a6" (UID: "3176c8ca-243b-4ad6-8b53-845e9a8843a6"). InnerVolumeSpecName "kube-api-access-x6zj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:50:14.741457 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:14.741376    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6zj6\" (UniqueName: \"kubernetes.io/projected/3176c8ca-243b-4ad6-8b53-845e9a8843a6-kube-api-access-x6zj6\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:50:14.741457 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:14.741418    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3176c8ca-243b-4ad6-8b53-845e9a8843a6-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:50:14.741457 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:14.741432    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3176c8ca-243b-4ad6-8b53-845e9a8843a6-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:50:14.741457 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:14.741445    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3176c8ca-243b-4ad6-8b53-845e9a8843a6-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:50:14.791371 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:14.791326    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"
Oct 30 00:50:14.791757 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:14.791728    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"
Oct 30 00:50:14.792923 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:14.792902    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"
Oct 30 00:50:14.972149 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:14.972093    4393 generic.go:356] "Generic (PLEG): container finished" podID="3176c8ca-243b-4ad6-8b53-845e9a8843a6" containerID="cba7af3bdf0616ad7970c58f1e1b427d3c0bcdc9d69b2579f33cf58d403aacbd" exitCode=0
Oct 30 00:50:14.972330 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:14.972158    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj" event={"ID":"3176c8ca-243b-4ad6-8b53-845e9a8843a6","Type":"ContainerDied","Data":"cba7af3bdf0616ad7970c58f1e1b427d3c0bcdc9d69b2579f33cf58d403aacbd"}
Oct 30 00:50:14.972330 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:14.972132    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj"
Oct 30 00:50:14.972330 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:14.972217    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj" event={"ID":"3176c8ca-243b-4ad6-8b53-845e9a8843a6","Type":"ContainerDied","Data":"edf73212dd05f3d89209a62f6ad4cd49d577688e8f2732dabb170b9778172d18"}
Oct 30 00:50:14.972330 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:14.972247    4393 scope.go:122] "RemoveContainer" containerID="cba7af3bdf0616ad7970c58f1e1b427d3c0bcdc9d69b2579f33cf58d403aacbd"
Oct 30 00:50:14.984086 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:14.984064    4393 scope.go:122] "RemoveContainer" containerID="cba7af3bdf0616ad7970c58f1e1b427d3c0bcdc9d69b2579f33cf58d403aacbd"
Oct 30 00:50:14.984354 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:50:14.984335    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cba7af3bdf0616ad7970c58f1e1b427d3c0bcdc9d69b2579f33cf58d403aacbd\": container with ID starting with cba7af3bdf0616ad7970c58f1e1b427d3c0bcdc9d69b2579f33cf58d403aacbd not found: ID does not exist" containerID="cba7af3bdf0616ad7970c58f1e1b427d3c0bcdc9d69b2579f33cf58d403aacbd"
Oct 30 00:50:14.984430 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:14.984369    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cba7af3bdf0616ad7970c58f1e1b427d3c0bcdc9d69b2579f33cf58d403aacbd"} err="failed to get container status \"cba7af3bdf0616ad7970c58f1e1b427d3c0bcdc9d69b2579f33cf58d403aacbd\": rpc error: code = NotFound desc = could not find container \"cba7af3bdf0616ad7970c58f1e1b427d3c0bcdc9d69b2579f33cf58d403aacbd\": container with ID starting with cba7af3bdf0616ad7970c58f1e1b427d3c0bcdc9d69b2579f33cf58d403aacbd not found: ID does not exist"
Oct 30 00:50:15.127300 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:15.127266    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj"]
Oct 30 00:50:15.145805 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:15.128016    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"]
Oct 30 00:50:15.145805 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:15.128248    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944" podUID="93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a" containerName="istio-proxy" containerID="cri-o://345b06fbc7d39f68db26f3734592cf0cf08074cdc9b5c6a7678d3a5010b32040" gracePeriod=30
Oct 30 00:50:15.145805 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:15.142551    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cab9990d-67856cd894-mzbtj"]
Oct 30 00:50:15.503032 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:15.502926    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-cab9990d-75bddb75b6-nsjnn" podUID="09c71ec7-db60-4886-9e2b-878a1bbf659b" containerName="payload-processing" containerID="cri-o://e1095483026025b25affd4df9e3c36e648aac1168eb9b5eb9af90e19694f709f" gracePeriod=25
Oct 30 00:50:15.540861 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:15.540820    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cab9990d-76b4b665cf-75mnn" podUID="a802164a-cc5f-4c52-9047-e69c63002b66" containerName="payload-pre-processing" containerID="cri-o://6346739217c1ebff606995a64b1f0e36fd01da9174637210e1b174ee4f43f84f" gracePeriod=25
Oct 30 00:50:15.740014 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:15.739976    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="125e2185-b66d-4853-94b5-1961ed27fc63" path="/var/lib/kubelet/pods/125e2185-b66d-4853-94b5-1961ed27fc63/volumes"
Oct 30 00:50:15.740576 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:15.740549    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3176c8ca-243b-4ad6-8b53-845e9a8843a6" path="/var/lib/kubelet/pods/3176c8ca-243b-4ad6-8b53-845e9a8843a6/volumes"
Oct 30 00:50:15.741232 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:15.741146    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="966264d7-a7c5-41c0-9c7e-dafd22a3d2ab" path="/var/lib/kubelet/pods/966264d7-a7c5-41c0-9c7e-dafd22a3d2ab/volumes"
Oct 30 00:50:15.832706 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:15.832678    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cab9990d-75bddb75b6-nsjnn"
Oct 30 00:50:15.840711 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:15.840677    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09c71ec7-db60-4886-9e2b-878a1bbf659b-kube-api-access-cxrlw\" (UniqueName: \"kubernetes.io/projected/09c71ec7-db60-4886-9e2b-878a1bbf659b-kube-api-access-cxrlw\") pod \"09c71ec7-db60-4886-9e2b-878a1bbf659b\" (UID: \"09c71ec7-db60-4886-9e2b-878a1bbf659b\") "
Oct 30 00:50:15.840863 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:15.840821    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09c71ec7-db60-4886-9e2b-878a1bbf659b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09c71ec7-db60-4886-9e2b-878a1bbf659b-plugins-config-volume\") pod \"09c71ec7-db60-4886-9e2b-878a1bbf659b\" (UID: \"09c71ec7-db60-4886-9e2b-878a1bbf659b\") "
Oct 30 00:50:15.841132 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:15.841110    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09c71ec7-db60-4886-9e2b-878a1bbf659b-plugins-config-volume" pod "09c71ec7-db60-4886-9e2b-878a1bbf659b" (UID: "09c71ec7-db60-4886-9e2b-878a1bbf659b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:50:15.841426 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:15.841410    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09c71ec7-db60-4886-9e2b-878a1bbf659b-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:50:15.843091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:15.843068    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c71ec7-db60-4886-9e2b-878a1bbf659b-kube-api-access-cxrlw" pod "09c71ec7-db60-4886-9e2b-878a1bbf659b" (UID: "09c71ec7-db60-4886-9e2b-878a1bbf659b"). InnerVolumeSpecName "kube-api-access-cxrlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:50:15.905376 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:15.905342    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cab9990d-76b4b665cf-75mnn"
Oct 30 00:50:15.942886 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:15.942850    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a802164a-cc5f-4c52-9047-e69c63002b66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a802164a-cc5f-4c52-9047-e69c63002b66-plugins-config-volume\") pod \"a802164a-cc5f-4c52-9047-e69c63002b66\" (UID: \"a802164a-cc5f-4c52-9047-e69c63002b66\") "
Oct 30 00:50:15.943048 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:15.942968    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a802164a-cc5f-4c52-9047-e69c63002b66-kube-api-access-jrdgk\" (UniqueName: \"kubernetes.io/projected/a802164a-cc5f-4c52-9047-e69c63002b66-kube-api-access-jrdgk\") pod \"a802164a-cc5f-4c52-9047-e69c63002b66\" (UID: \"a802164a-cc5f-4c52-9047-e69c63002b66\") "
Oct 30 00:50:15.943216 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:15.943192    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a802164a-cc5f-4c52-9047-e69c63002b66-plugins-config-volume" pod "a802164a-cc5f-4c52-9047-e69c63002b66" (UID: "a802164a-cc5f-4c52-9047-e69c63002b66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:50:15.943842 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:15.943816    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a802164a-cc5f-4c52-9047-e69c63002b66-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:50:15.943965 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:15.943850    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxrlw\" (UniqueName: \"kubernetes.io/projected/09c71ec7-db60-4886-9e2b-878a1bbf659b-kube-api-access-cxrlw\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:50:15.945846 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:15.945822    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a802164a-cc5f-4c52-9047-e69c63002b66-kube-api-access-jrdgk" pod "a802164a-cc5f-4c52-9047-e69c63002b66" (UID: "a802164a-cc5f-4c52-9047-e69c63002b66"). InnerVolumeSpecName "kube-api-access-jrdgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:50:15.983538 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:15.983506    4393 generic.go:356] "Generic (PLEG): container finished" podID="09c71ec7-db60-4886-9e2b-878a1bbf659b" containerID="e1095483026025b25affd4df9e3c36e648aac1168eb9b5eb9af90e19694f709f" exitCode=0
Oct 30 00:50:15.983717 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:15.983552    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cab9990d-75bddb75b6-nsjnn"
Oct 30 00:50:15.983717 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:15.983590    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cab9990d-75bddb75b6-nsjnn" event={"ID":"09c71ec7-db60-4886-9e2b-878a1bbf659b","Type":"ContainerDied","Data":"e1095483026025b25affd4df9e3c36e648aac1168eb9b5eb9af90e19694f709f"}
Oct 30 00:50:15.983717 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:15.983649    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cab9990d-75bddb75b6-nsjnn" event={"ID":"09c71ec7-db60-4886-9e2b-878a1bbf659b","Type":"ContainerDied","Data":"09d48723f534770495d15d2b6b410a90fb38a27eaf02e47c575b91b720a663c8"}
Oct 30 00:50:15.983717 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:15.983678    4393 scope.go:122] "RemoveContainer" containerID="e1095483026025b25affd4df9e3c36e648aac1168eb9b5eb9af90e19694f709f"
Oct 30 00:50:15.987105 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:15.987077    4393 generic.go:356] "Generic (PLEG): container finished" podID="a802164a-cc5f-4c52-9047-e69c63002b66" containerID="6346739217c1ebff606995a64b1f0e36fd01da9174637210e1b174ee4f43f84f" exitCode=0
Oct 30 00:50:15.987226 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:15.987135    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cab9990d-76b4b665cf-75mnn"
Oct 30 00:50:15.987226 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:15.987184    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cab9990d-76b4b665cf-75mnn" event={"ID":"a802164a-cc5f-4c52-9047-e69c63002b66","Type":"ContainerDied","Data":"6346739217c1ebff606995a64b1f0e36fd01da9174637210e1b174ee4f43f84f"}
Oct 30 00:50:15.987226 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:15.987220    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cab9990d-76b4b665cf-75mnn" event={"ID":"a802164a-cc5f-4c52-9047-e69c63002b66","Type":"ContainerDied","Data":"79466099823ffa1232edacb5e68783ce29aab3b3cede0a6fa01ee5e89e20fdeb"}
Oct 30 00:50:15.995791 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:15.995754    4393 scope.go:122] "RemoveContainer" containerID="e1095483026025b25affd4df9e3c36e648aac1168eb9b5eb9af90e19694f709f"
Oct 30 00:50:15.996116 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:50:15.996060    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1095483026025b25affd4df9e3c36e648aac1168eb9b5eb9af90e19694f709f\": container with ID starting with e1095483026025b25affd4df9e3c36e648aac1168eb9b5eb9af90e19694f709f not found: ID does not exist" containerID="e1095483026025b25affd4df9e3c36e648aac1168eb9b5eb9af90e19694f709f"
Oct 30 00:50:15.996116 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:15.996102    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1095483026025b25affd4df9e3c36e648aac1168eb9b5eb9af90e19694f709f"} err="failed to get container status \"e1095483026025b25affd4df9e3c36e648aac1168eb9b5eb9af90e19694f709f\": rpc error: code = NotFound desc = could not find container \"e1095483026025b25affd4df9e3c36e648aac1168eb9b5eb9af90e19694f709f\": container with ID starting with e1095483026025b25affd4df9e3c36e648aac1168eb9b5eb9af90e19694f709f not found: ID does not exist"
Oct 30 00:50:15.996310 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:15.996120    4393 scope.go:122] "RemoveContainer" containerID="6346739217c1ebff606995a64b1f0e36fd01da9174637210e1b174ee4f43f84f"
Oct 30 00:50:16.010759 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:16.010735    4393 scope.go:122] "RemoveContainer" containerID="6346739217c1ebff606995a64b1f0e36fd01da9174637210e1b174ee4f43f84f"
Oct 30 00:50:16.011111 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:50:16.011088    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6346739217c1ebff606995a64b1f0e36fd01da9174637210e1b174ee4f43f84f\": container with ID starting with 6346739217c1ebff606995a64b1f0e36fd01da9174637210e1b174ee4f43f84f not found: ID does not exist" containerID="6346739217c1ebff606995a64b1f0e36fd01da9174637210e1b174ee4f43f84f"
Oct 30 00:50:16.011166 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:16.011120    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6346739217c1ebff606995a64b1f0e36fd01da9174637210e1b174ee4f43f84f"} err="failed to get container status \"6346739217c1ebff606995a64b1f0e36fd01da9174637210e1b174ee4f43f84f\": rpc error: code = NotFound desc = could not find container \"6346739217c1ebff606995a64b1f0e36fd01da9174637210e1b174ee4f43f84f\": container with ID starting with 6346739217c1ebff606995a64b1f0e36fd01da9174637210e1b174ee4f43f84f not found: ID does not exist"
Oct 30 00:50:16.045319 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:16.045228    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrdgk\" (UniqueName: \"kubernetes.io/projected/a802164a-cc5f-4c52-9047-e69c63002b66-kube-api-access-jrdgk\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:50:16.050146 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:16.050119    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cab9990d-75bddb75b6-nsjnn"]
Oct 30 00:50:16.056060 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:16.056034    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cab9990d-75bddb75b6-nsjnn"]
Oct 30 00:50:16.090568 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:16.090534    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cab9990d-76b4b665cf-75mnn"]
Oct 30 00:50:16.095967 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:16.095944    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cab9990d-76b4b665cf-75mnn"]
Oct 30 00:50:16.520977 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:16.520947    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-751d6789-7fb4c76b79-l6pp8"]
Oct 30 00:50:16.521637 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:16.521623    4393 cpu_manager.go:414] "containerMap: removing container" podUID="09c71ec7-db60-4886-9e2b-878a1bbf659b" containerName="payload-processing"
Oct 30 00:50:16.521682 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:16.521639    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09c71ec7-db60-4886-9e2b-878a1bbf659b" containerName="payload-processing"
Oct 30 00:50:16.521682 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:16.521653    4393 cpu_manager.go:414] "containerMap: removing container" podUID="966264d7-a7c5-41c0-9c7e-dafd22a3d2ab" containerName="payload-pre-processing"
Oct 30 00:50:16.521682 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:16.521659    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="966264d7-a7c5-41c0-9c7e-dafd22a3d2ab" containerName="payload-pre-processing"
Oct 30 00:50:16.521834 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:16.521683    4393 cpu_manager.go:414] "containerMap: removing container" podUID="a802164a-cc5f-4c52-9047-e69c63002b66" containerName="payload-pre-processing"
Oct 30 00:50:16.521834 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:16.521688    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a802164a-cc5f-4c52-9047-e69c63002b66" containerName="payload-pre-processing"
Oct 30 00:50:16.521834 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:16.521709    4393 cpu_manager.go:414] "containerMap: removing container" podUID="125e2185-b66d-4853-94b5-1961ed27fc63" containerName="payload-processing"
Oct 30 00:50:16.521834 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:16.521714    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="125e2185-b66d-4853-94b5-1961ed27fc63" containerName="payload-processing"
Oct 30 00:50:16.521834 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:16.521722    4393 cpu_manager.go:414] "containerMap: removing container" podUID="b71b75d8-eb01-4748-a17f-2cc7b69f8614" containerName="maas-api"
Oct 30 00:50:16.521834 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:16.521727    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b71b75d8-eb01-4748-a17f-2cc7b69f8614" containerName="maas-api"
Oct 30 00:50:16.521834 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:16.521742    4393 cpu_manager.go:414] "containerMap: removing container" podUID="3176c8ca-243b-4ad6-8b53-845e9a8843a6" containerName="maas-api"
Oct 30 00:50:16.521834 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:16.521746    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3176c8ca-243b-4ad6-8b53-845e9a8843a6" containerName="maas-api"
Oct 30 00:50:16.522067 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:16.521858    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="125e2185-b66d-4853-94b5-1961ed27fc63" containerName="payload-processing"
Oct 30 00:50:16.522067 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:16.521867    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="b71b75d8-eb01-4748-a17f-2cc7b69f8614" containerName="maas-api"
Oct 30 00:50:16.522067 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:16.521874    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="3176c8ca-243b-4ad6-8b53-845e9a8843a6" containerName="maas-api"
Oct 30 00:50:16.522067 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:16.521881    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="09c71ec7-db60-4886-9e2b-878a1bbf659b" containerName="payload-processing"
Oct 30 00:50:16.522067 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:16.521886    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="966264d7-a7c5-41c0-9c7e-dafd22a3d2ab" containerName="payload-pre-processing"
Oct 30 00:50:16.522067 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:16.521895    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="a802164a-cc5f-4c52-9047-e69c63002b66" containerName="payload-pre-processing"
Oct 30 00:50:16.527204 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:16.527185    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-751d6789-7fb4c76b79-l6pp8"
Oct 30 00:50:16.587808 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:16.585112    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e8b8136-7f22-47fb-8c7c-bd11e4bb6281-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-751d6789-7fb4c76b79-l6pp8\" (UID: \"8e8b8136-7f22-47fb-8c7c-bd11e4bb6281\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-751d6789-7fb4c76b79-l6pp8"
Oct 30 00:50:16.587808 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:16.585425    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngkv8\" (UniqueName: \"kubernetes.io/projected/8e8b8136-7f22-47fb-8c7c-bd11e4bb6281-kube-api-access-ngkv8\") pod \"payload-pre-processing-e2e-ait-751d6789-7fb4c76b79-l6pp8\" (UID: \"8e8b8136-7f22-47fb-8c7c-bd11e4bb6281\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-751d6789-7fb4c76b79-l6pp8"
Oct 30 00:50:16.587808 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:16.587053    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-751d6789-dockercfg-bkj5d\""
Oct 30 00:50:16.587808 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:16.587317    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-751d6789\""
Oct 30 00:50:16.849803 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:16.849577    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-751d6789-7fb4c76b79-l6pp8"]
Oct 30 00:50:16.891823 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:16.890553    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e8b8136-7f22-47fb-8c7c-bd11e4bb6281-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-751d6789-7fb4c76b79-l6pp8\" (UID: \"8e8b8136-7f22-47fb-8c7c-bd11e4bb6281\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-751d6789-7fb4c76b79-l6pp8"
Oct 30 00:50:16.891823 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:16.891322    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngkv8\" (UniqueName: \"kubernetes.io/projected/8e8b8136-7f22-47fb-8c7c-bd11e4bb6281-kube-api-access-ngkv8\") pod \"payload-pre-processing-e2e-ait-751d6789-7fb4c76b79-l6pp8\" (UID: \"8e8b8136-7f22-47fb-8c7c-bd11e4bb6281\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-751d6789-7fb4c76b79-l6pp8"
Oct 30 00:50:16.899115 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:16.898799    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e8b8136-7f22-47fb-8c7c-bd11e4bb6281-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-751d6789-7fb4c76b79-l6pp8\" (UID: \"8e8b8136-7f22-47fb-8c7c-bd11e4bb6281\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-751d6789-7fb4c76b79-l6pp8"
Oct 30 00:50:16.985332 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:16.984334    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngkv8\" (UniqueName: \"kubernetes.io/projected/8e8b8136-7f22-47fb-8c7c-bd11e4bb6281-kube-api-access-ngkv8\") pod \"payload-pre-processing-e2e-ait-751d6789-7fb4c76b79-l6pp8\" (UID: \"8e8b8136-7f22-47fb-8c7c-bd11e4bb6281\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-751d6789-7fb4c76b79-l6pp8"
Oct 30 00:50:17.045987 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:17.045958    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-751d6789-7bd9589f56-8vx54"]
Oct 30 00:50:17.054991 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:17.054959    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-751d6789-7bd9589f56-8vx54"
Oct 30 00:50:17.057923 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:17.057894    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-751d6789-7bd9589f56-8vx54"]
Oct 30 00:50:17.141436 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:17.141405    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-751d6789-7fb4c76b79-l6pp8"
Oct 30 00:50:17.148108 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:17.148081    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5f99387-2f87-423c-89ca-1f7676f8c4c6-plugins-config-volume\") pod \"payload-processing-e2e-ait-751d6789-7bd9589f56-8vx54\" (UID: \"f5f99387-2f87-423c-89ca-1f7676f8c4c6\") " pod="openshift-ingress/payload-processing-e2e-ait-751d6789-7bd9589f56-8vx54"
Oct 30 00:50:17.148226 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:17.148202    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6bdr\" (UniqueName: \"kubernetes.io/projected/f5f99387-2f87-423c-89ca-1f7676f8c4c6-kube-api-access-q6bdr\") pod \"payload-processing-e2e-ait-751d6789-7bd9589f56-8vx54\" (UID: \"f5f99387-2f87-423c-89ca-1f7676f8c4c6\") " pod="openshift-ingress/payload-processing-e2e-ait-751d6789-7bd9589f56-8vx54"
Oct 30 00:50:17.249741 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:17.249705    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5f99387-2f87-423c-89ca-1f7676f8c4c6-plugins-config-volume\") pod \"payload-processing-e2e-ait-751d6789-7bd9589f56-8vx54\" (UID: \"f5f99387-2f87-423c-89ca-1f7676f8c4c6\") " pod="openshift-ingress/payload-processing-e2e-ait-751d6789-7bd9589f56-8vx54"
Oct 30 00:50:17.249942 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:17.249859    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6bdr\" (UniqueName: \"kubernetes.io/projected/f5f99387-2f87-423c-89ca-1f7676f8c4c6-kube-api-access-q6bdr\") pod \"payload-processing-e2e-ait-751d6789-7bd9589f56-8vx54\" (UID: \"f5f99387-2f87-423c-89ca-1f7676f8c4c6\") " pod="openshift-ingress/payload-processing-e2e-ait-751d6789-7bd9589f56-8vx54"
Oct 30 00:50:17.250744 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:17.250717    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5f99387-2f87-423c-89ca-1f7676f8c4c6-plugins-config-volume\") pod \"payload-processing-e2e-ait-751d6789-7bd9589f56-8vx54\" (UID: \"f5f99387-2f87-423c-89ca-1f7676f8c4c6\") " pod="openshift-ingress/payload-processing-e2e-ait-751d6789-7bd9589f56-8vx54"
Oct 30 00:50:17.273004 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:17.272976    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6bdr\" (UniqueName: \"kubernetes.io/projected/f5f99387-2f87-423c-89ca-1f7676f8c4c6-kube-api-access-q6bdr\") pod \"payload-processing-e2e-ait-751d6789-7bd9589f56-8vx54\" (UID: \"f5f99387-2f87-423c-89ca-1f7676f8c4c6\") " pod="openshift-ingress/payload-processing-e2e-ait-751d6789-7bd9589f56-8vx54"
Oct 30 00:50:17.370667 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:17.370635    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-751d6789-7bd9589f56-8vx54"
Oct 30 00:50:17.455242 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:17.455211    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-751d6789-7fb4c76b79-l6pp8"]
Oct 30 00:50:17.462979 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:50:17.462951    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e8b8136_7f22_47fb_8c7c_bd11e4bb6281.slice/crio-06e7307ae1eb42e286747a14a5847d8e667fc0d513dbdc544aff61acedc7a33a WatchSource:0}: Error finding container 06e7307ae1eb42e286747a14a5847d8e667fc0d513dbdc544aff61acedc7a33a: Status 404 returned error can't find the container with id 06e7307ae1eb42e286747a14a5847d8e667fc0d513dbdc544aff61acedc7a33a
Oct 30 00:50:17.679386 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:50:17.679354    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5f99387_2f87_423c_89ca_1f7676f8c4c6.slice/crio-d15289e9e0a9f43fbd061e5ba0c44b764404867bbf643c12c4d4e6601f2a045c WatchSource:0}: Error finding container d15289e9e0a9f43fbd061e5ba0c44b764404867bbf643c12c4d4e6601f2a045c: Status 404 returned error can't find the container with id d15289e9e0a9f43fbd061e5ba0c44b764404867bbf643c12c4d4e6601f2a045c
Oct 30 00:50:17.679908 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:17.679886    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-751d6789-7bd9589f56-8vx54"]
Oct 30 00:50:17.728421 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:17.728390    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09c71ec7-db60-4886-9e2b-878a1bbf659b" path="/var/lib/kubelet/pods/09c71ec7-db60-4886-9e2b-878a1bbf659b/volumes"
Oct 30 00:50:17.729015 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:17.728996    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a802164a-cc5f-4c52-9047-e69c63002b66" path="/var/lib/kubelet/pods/a802164a-cc5f-4c52-9047-e69c63002b66/volumes"
Oct 30 00:50:18.066317 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:18.066225    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-751d6789-7bd9589f56-8vx54" event={"ID":"f5f99387-2f87-423c-89ca-1f7676f8c4c6","Type":"ContainerStarted","Data":"d607e65dee32cc7254eec77807128c933a0b885476670294fa63a3b80800ca8d"}
Oct 30 00:50:18.066317 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:18.066287    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-751d6789-7bd9589f56-8vx54" event={"ID":"f5f99387-2f87-423c-89ca-1f7676f8c4c6","Type":"ContainerStarted","Data":"d15289e9e0a9f43fbd061e5ba0c44b764404867bbf643c12c4d4e6601f2a045c"}
Oct 30 00:50:18.066551 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:18.066320    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-751d6789-7bd9589f56-8vx54"
Oct 30 00:50:18.068078 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:18.068044    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-751d6789-7fb4c76b79-l6pp8" event={"ID":"8e8b8136-7f22-47fb-8c7c-bd11e4bb6281","Type":"ContainerStarted","Data":"2306f1b6df9ca352242ba431d190507c96557c46fb1937c19dd5a97d48381f96"}
Oct 30 00:50:18.068218 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:18.068101    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-751d6789-7fb4c76b79-l6pp8" event={"ID":"8e8b8136-7f22-47fb-8c7c-bd11e4bb6281","Type":"ContainerStarted","Data":"06e7307ae1eb42e286747a14a5847d8e667fc0d513dbdc544aff61acedc7a33a"}
Oct 30 00:50:18.068218 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:18.068132    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-751d6789-7fb4c76b79-l6pp8"
Oct 30 00:50:18.106543 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:18.106486    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-751d6789-7bd9589f56-8vx54" podStartSLOduration=2.106471565 podStartE2EDuration="2.106471565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:18.104738364 +0000 UTC m=+1108.492414752" watchObservedRunningTime="2026-10-30 00:50:18.106471565 +0000 UTC m=+1108.494147953"
Oct 30 00:50:18.144649 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:18.144600    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-751d6789-7fb4c76b79-l6pp8" podStartSLOduration=2.144585844 podStartE2EDuration="2.144585844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:18.143717404 +0000 UTC m=+1108.531393793" watchObservedRunningTime="2026-10-30 00:50:18.144585844 +0000 UTC m=+1108.532262255"
Oct 30 00:50:20.413525 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.413156    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"
Oct 30 00:50:20.586515 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.586480    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-istio-podinfo\") pod \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\" (UID: \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\") "
Oct 30 00:50:20.586704 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.586608    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-workload-socket\") pod \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\" (UID: \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\") "
Oct 30 00:50:20.586704 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.586649    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-credential-socket\") pod \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\" (UID: \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\") "
Oct 30 00:50:20.586704 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.586674    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-istio-envoy\") pod \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\" (UID: \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\") "
Oct 30 00:50:20.586901 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.586835    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-istiod-ca-cert\") pod \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\" (UID: \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\") "
Oct 30 00:50:20.586955 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.586931    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-kube-api-access-jgzxt\" (UniqueName: \"kubernetes.io/projected/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-kube-api-access-jgzxt\") pod \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\" (UID: \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\") "
Oct 30 00:50:20.587007 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.586966    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-istio-token\" (UniqueName: \"kubernetes.io/projected/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-istio-token\") pod \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\" (UID: \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\") "
Oct 30 00:50:20.587062 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.587015    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-workload-certs\") pod \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\" (UID: \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\") "
Oct 30 00:50:20.587062 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.587024    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-workload-socket" pod "93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a" (UID: "93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:50:20.587062 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.587035    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-credential-socket" pod "93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a" (UID: "93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:50:20.587062 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.587054    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-istio-data\") pod \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\" (UID: \"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a\") "
Oct 30 00:50:20.587306 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.587223    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-istiod-ca-cert" pod "93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a" (UID: "93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:50:20.587306 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.587287    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-workload-certs" pod "93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a" (UID: "93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:50:20.587426 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.587409    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-istio-data" pod "93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a" (UID: "93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:50:20.588028 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.587969    4393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-istiod-ca-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:50:20.588028 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.587994    4393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-workload-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:50:20.588028 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.588007    4393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-istio-data\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:50:20.588028 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.588020    4393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-workload-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:50:20.588028 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.588032    4393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-credential-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:50:20.589600 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.589576    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-istio-envoy" pod "93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a" (UID: "93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:50:20.589708 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.589691    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-istio-podinfo" pod "93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a" (UID: "93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:50:20.589754 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.589715    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-istio-token" pod "93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a" (UID: "93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:50:20.589754 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.589744    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-kube-api-access-jgzxt" pod "93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a" (UID: "93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a"). InnerVolumeSpecName "kube-api-access-jgzxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:50:20.606975 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.606952    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"]
Oct 30 00:50:20.607650 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.607637    4393 cpu_manager.go:414] "containerMap: removing container" podUID="93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a" containerName="istio-proxy"
Oct 30 00:50:20.607696 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.607651    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a" containerName="istio-proxy"
Oct 30 00:50:20.607796 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.607771    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a" containerName="istio-proxy"
Oct 30 00:50:20.611520 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.611498    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"
Oct 30 00:50:20.644875 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.644846    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"]
Oct 30 00:50:20.689486 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.689410    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgzxt\" (UniqueName: \"kubernetes.io/projected/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-kube-api-access-jgzxt\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:50:20.689486 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.689437    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:50:20.689486 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.689447    4393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-istio-podinfo\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:50:20.689486 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.689455    4393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a-istio-envoy\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:50:20.791184 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.791145    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2bt8\" (UniqueName: \"kubernetes.io/projected/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-kube-api-access-x2bt8\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8\" (UID: \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"
Oct 30 00:50:20.791184 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.791188    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-istio-envoy\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8\" (UID: \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"
Oct 30 00:50:20.791426 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.791318    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-workload-socket\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8\" (UID: \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"
Oct 30 00:50:20.791426 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.791337    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-istio-podinfo\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8\" (UID: \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"
Oct 30 00:50:20.791529 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.791441    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-workload-certs\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8\" (UID: \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"
Oct 30 00:50:20.791529 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.791503    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-istio-data\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8\" (UID: \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"
Oct 30 00:50:20.791623 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.791608    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-credential-socket\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8\" (UID: \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"
Oct 30 00:50:20.791687 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.791670    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-istiod-ca-cert\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8\" (UID: \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"
Oct 30 00:50:20.791759 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.791736    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-istio-token\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8\" (UID: \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"
Oct 30 00:50:20.892878 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.892845    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-istio-data\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8\" (UID: \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"
Oct 30 00:50:20.893031 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.892926    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-credential-socket\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8\" (UID: \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"
Oct 30 00:50:20.893031 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.893015    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-istiod-ca-cert\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8\" (UID: \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"
Oct 30 00:50:20.893120 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.893044    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-istio-token\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8\" (UID: \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"
Oct 30 00:50:20.893151 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.893125    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2bt8\" (UniqueName: \"kubernetes.io/projected/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-kube-api-access-x2bt8\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8\" (UID: \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"
Oct 30 00:50:20.893189 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.893155    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-istio-envoy\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8\" (UID: \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"
Oct 30 00:50:20.893364 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.893229    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-workload-socket\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8\" (UID: \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"
Oct 30 00:50:20.893364 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.893283    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-credential-socket\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8\" (UID: \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"
Oct 30 00:50:20.893364 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.893302    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-istio-podinfo\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8\" (UID: \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"
Oct 30 00:50:20.893364 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.893341    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-istio-data\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8\" (UID: \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"
Oct 30 00:50:20.893613 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.893412    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-workload-certs\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8\" (UID: \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"
Oct 30 00:50:20.893722 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.893702    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-workload-socket\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8\" (UID: \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"
Oct 30 00:50:20.893972 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.893928    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-workload-certs\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8\" (UID: \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"
Oct 30 00:50:20.893972 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.893952    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-istiod-ca-cert\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8\" (UID: \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"
Oct 30 00:50:20.895925 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.895903    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-istio-envoy\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8\" (UID: \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"
Oct 30 00:50:20.896110 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.896091    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-istio-podinfo\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8\" (UID: \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"
Oct 30 00:50:20.917345 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.917315    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-istio-token\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8\" (UID: \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"
Oct 30 00:50:20.917512 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.917491    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2bt8\" (UniqueName: \"kubernetes.io/projected/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-kube-api-access-x2bt8\") pod \"e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8\" (UID: \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\") " pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"
Oct 30 00:50:20.935084 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:20.935063    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"
Oct 30 00:50:21.093883 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:21.093835    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944" event={"ID":"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a","Type":"ContainerDied","Data":"345b06fbc7d39f68db26f3734592cf0cf08074cdc9b5c6a7678d3a5010b32040"}
Oct 30 00:50:21.093883 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:21.093770    4393 generic.go:356] "Generic (PLEG): container finished" podID="93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a" containerID="345b06fbc7d39f68db26f3734592cf0cf08074cdc9b5c6a7678d3a5010b32040" exitCode=0
Oct 30 00:50:21.094082 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:21.093902    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944" event={"ID":"93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a","Type":"ContainerDied","Data":"6d1f5c29e6cb0836a9fe7b08a80f6d00ff966f680a823492167d776c73cc078b"}
Oct 30 00:50:21.094082 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:21.093924    4393 scope.go:122] "RemoveContainer" containerID="345b06fbc7d39f68db26f3734592cf0cf08074cdc9b5c6a7678d3a5010b32040"
Oct 30 00:50:21.094082 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:21.093860    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"
Oct 30 00:50:21.108602 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:21.108584    4393 scope.go:122] "RemoveContainer" containerID="345b06fbc7d39f68db26f3734592cf0cf08074cdc9b5c6a7678d3a5010b32040"
Oct 30 00:50:21.108906 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:50:21.108886    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"345b06fbc7d39f68db26f3734592cf0cf08074cdc9b5c6a7678d3a5010b32040\": container with ID starting with 345b06fbc7d39f68db26f3734592cf0cf08074cdc9b5c6a7678d3a5010b32040 not found: ID does not exist" containerID="345b06fbc7d39f68db26f3734592cf0cf08074cdc9b5c6a7678d3a5010b32040"
Oct 30 00:50:21.108968 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:21.108914    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"345b06fbc7d39f68db26f3734592cf0cf08074cdc9b5c6a7678d3a5010b32040"} err="failed to get container status \"345b06fbc7d39f68db26f3734592cf0cf08074cdc9b5c6a7678d3a5010b32040\": rpc error: code = NotFound desc = could not find container \"345b06fbc7d39f68db26f3734592cf0cf08074cdc9b5c6a7678d3a5010b32040\": container with ID starting with 345b06fbc7d39f68db26f3734592cf0cf08074cdc9b5c6a7678d3a5010b32040 not found: ID does not exist"
Oct 30 00:50:21.148447 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:21.148417    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"]
Oct 30 00:50:21.160036 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:21.160002    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-cab9990d-gw-openshift-default-77d8f976f7-vb944"]
Oct 30 00:50:21.261829 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:50:21.261800    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb8b407b_2bc7_4457_814b_31ffbdffa0c2.slice/crio-9db81db0bf52836e720fc46c0e04ed05053899f6e0f70e0419134a8a340a8906 WatchSource:0}: Error finding container 9db81db0bf52836e720fc46c0e04ed05053899f6e0f70e0419134a8a340a8906: Status 404 returned error can't find the container with id 9db81db0bf52836e720fc46c0e04ed05053899f6e0f70e0419134a8a340a8906
Oct 30 00:50:21.262091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:21.262059    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"]
Oct 30 00:50:21.265150 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:21.265119    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:50:21.265259 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:21.265180    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:50:21.265259 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:21.265211    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:50:21.727839 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:21.727807    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a" path="/var/lib/kubelet/pods/93b2c8be-8f15-4c5b-a9ee-22ac611c1e8a/volumes"
Oct 30 00:50:21.863106 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:21.863073    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-be862c05-c76879479-bfz2h"]
Oct 30 00:50:21.867910 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:21.867887    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-be862c05-c76879479-bfz2h"
Oct 30 00:50:21.883088 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:21.883061    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-be862c05-dockercfg-hffsf\""
Oct 30 00:50:21.883088 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:21.883061    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-be862c05\""
Oct 30 00:50:21.888071 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:21.888049    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-be862c05-c76879479-bfz2h"]
Oct 30 00:50:21.936858 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:21.928369    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e75c787-c583-464d-b087-be6f84901779-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-be862c05-c76879479-bfz2h\" (UID: \"8e75c787-c583-464d-b087-be6f84901779\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-be862c05-c76879479-bfz2h"
Oct 30 00:50:21.936858 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:21.928642    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcqnf\" (UniqueName: \"kubernetes.io/projected/8e75c787-c583-464d-b087-be6f84901779-kube-api-access-mcqnf\") pod \"payload-pre-processing-e2e-derive-be862c05-c76879479-bfz2h\" (UID: \"8e75c787-c583-464d-b087-be6f84901779\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-be862c05-c76879479-bfz2h"
Oct 30 00:50:22.105889 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:22.092712    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcqnf\" (UniqueName: \"kubernetes.io/projected/8e75c787-c583-464d-b087-be6f84901779-kube-api-access-mcqnf\") pod \"payload-pre-processing-e2e-derive-be862c05-c76879479-bfz2h\" (UID: \"8e75c787-c583-464d-b087-be6f84901779\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-be862c05-c76879479-bfz2h"
Oct 30 00:50:22.105889 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:22.092889    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e75c787-c583-464d-b087-be6f84901779-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-be862c05-c76879479-bfz2h\" (UID: \"8e75c787-c583-464d-b087-be6f84901779\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-be862c05-c76879479-bfz2h"
Oct 30 00:50:22.105889 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:22.093925    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e75c787-c583-464d-b087-be6f84901779-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-be862c05-c76879479-bfz2h\" (UID: \"8e75c787-c583-464d-b087-be6f84901779\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-be862c05-c76879479-bfz2h"
Oct 30 00:50:22.138918 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:22.138832    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-be862c05-6b54db7858-9bx9r"]
Oct 30 00:50:22.304834 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:22.298967    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-be862c05-6b54db7858-9bx9r"
Oct 30 00:50:22.364891 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:22.306855    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcqnf\" (UniqueName: \"kubernetes.io/projected/8e75c787-c583-464d-b087-be6f84901779-kube-api-access-mcqnf\") pod \"payload-pre-processing-e2e-derive-be862c05-c76879479-bfz2h\" (UID: \"8e75c787-c583-464d-b087-be6f84901779\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-be862c05-c76879479-bfz2h"
Oct 30 00:50:22.528355 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:22.519448    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqrlc\" (UniqueName: \"kubernetes.io/projected/db72d8fa-fc86-4654-b097-7d6a920a4d72-kube-api-access-vqrlc\") pod \"payload-processing-e2e-derive-be862c05-6b54db7858-9bx9r\" (UID: \"db72d8fa-fc86-4654-b097-7d6a920a4d72\") " pod="openshift-ingress/payload-processing-e2e-derive-be862c05-6b54db7858-9bx9r"
Oct 30 00:50:22.528355 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:22.519701    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db72d8fa-fc86-4654-b097-7d6a920a4d72-plugins-config-volume\") pod \"payload-processing-e2e-derive-be862c05-6b54db7858-9bx9r\" (UID: \"db72d8fa-fc86-4654-b097-7d6a920a4d72\") " pod="openshift-ingress/payload-processing-e2e-derive-be862c05-6b54db7858-9bx9r"
Oct 30 00:50:22.572812 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:22.540899    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-be862c05-c76879479-bfz2h"
Oct 30 00:50:22.658854 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:22.645015    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqrlc\" (UniqueName: \"kubernetes.io/projected/db72d8fa-fc86-4654-b097-7d6a920a4d72-kube-api-access-vqrlc\") pod \"payload-processing-e2e-derive-be862c05-6b54db7858-9bx9r\" (UID: \"db72d8fa-fc86-4654-b097-7d6a920a4d72\") " pod="openshift-ingress/payload-processing-e2e-derive-be862c05-6b54db7858-9bx9r"
Oct 30 00:50:22.658854 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:22.645191    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db72d8fa-fc86-4654-b097-7d6a920a4d72-plugins-config-volume\") pod \"payload-processing-e2e-derive-be862c05-6b54db7858-9bx9r\" (UID: \"db72d8fa-fc86-4654-b097-7d6a920a4d72\") " pod="openshift-ingress/payload-processing-e2e-derive-be862c05-6b54db7858-9bx9r"
Oct 30 00:50:22.658854 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:22.647088    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db72d8fa-fc86-4654-b097-7d6a920a4d72-plugins-config-volume\") pod \"payload-processing-e2e-derive-be862c05-6b54db7858-9bx9r\" (UID: \"db72d8fa-fc86-4654-b097-7d6a920a4d72\") " pod="openshift-ingress/payload-processing-e2e-derive-be862c05-6b54db7858-9bx9r"
Oct 30 00:50:22.738089 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:22.699590    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqrlc\" (UniqueName: \"kubernetes.io/projected/db72d8fa-fc86-4654-b097-7d6a920a4d72-kube-api-access-vqrlc\") pod \"payload-processing-e2e-derive-be862c05-6b54db7858-9bx9r\" (UID: \"db72d8fa-fc86-4654-b097-7d6a920a4d72\") " pod="openshift-ingress/payload-processing-e2e-derive-be862c05-6b54db7858-9bx9r"
Oct 30 00:50:22.765075 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:22.762938    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-be862c05-6b54db7858-9bx9r"]
Oct 30 00:50:22.910865 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:22.869572    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-be862c05-6b54db7858-9bx9r"
Oct 30 00:50:23.089846 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:23.065896    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q" podUID="46b01388-7da7-4e63-95f3-afc811bed013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:50:23.109524 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:23.107545    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x" podUID="28f3e4ff-5875-4e5d-8cab-60d1a5ba97be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:50:23.162043 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:23.161708    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8" event={"ID":"eb8b407b-2bc7-4457-814b-31ffbdffa0c2","Type":"ContainerStarted","Data":"4c5c8f964a98d054cfc3f0d40790fc145fc1ca59e5bb5632b1b362282cecf9d9"}
Oct 30 00:50:23.162043 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:23.161807    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8" event={"ID":"eb8b407b-2bc7-4457-814b-31ffbdffa0c2","Type":"ContainerStarted","Data":"9db81db0bf52836e720fc46c0e04ed05053899f6e0f70e0419134a8a340a8906"}
Oct 30 00:50:23.506860 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:23.505757    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8" podStartSLOduration=3.505737269 podStartE2EDuration="3.505737269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:23.222335266 +0000 UTC m=+1113.610011656" watchObservedRunningTime="2026-10-30 00:50:23.505737269 +0000 UTC m=+1113.893413660"
Oct 30 00:50:23.587810 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:23.585852    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-be862c05-c76879479-bfz2h"]
Oct 30 00:50:23.951754 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:23.951333    4393 patch_prober.go:29] interesting pod/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 00:50:23.951754 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:23.951411    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8" podUID="eb8b407b-2bc7-4457-814b-31ffbdffa0c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 00:50:23.979384 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:23.979359    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"
Oct 30 00:50:23.979655 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:23.979621    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-be862c05-6b54db7858-9bx9r"]
Oct 30 00:50:24.393878 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:24.393823    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-be862c05-6b54db7858-9bx9r" event={"ID":"db72d8fa-fc86-4654-b097-7d6a920a4d72","Type":"ContainerStarted","Data":"323059f4cbc40ff280eec38c0aa8d2c2bb727b37986bc25e1b630c11d7ad2616"}
Oct 30 00:50:24.420411 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:24.420380    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-be862c05-c76879479-bfz2h" event={"ID":"8e75c787-c583-464d-b087-be6f84901779","Type":"ContainerStarted","Data":"3ca5d88368eb972c7c7b481ab22790c18a7d221fff496afc67e37cdf2d20caba"}
Oct 30 00:50:24.936175 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:24.936141    4393 patch_prober.go:29] interesting pod/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 00:50:24.936404 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:24.936379    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8" podUID="eb8b407b-2bc7-4457-814b-31ffbdffa0c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 00:50:25.455928 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:25.455890    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-be862c05-6b54db7858-9bx9r" event={"ID":"db72d8fa-fc86-4654-b097-7d6a920a4d72","Type":"ContainerStarted","Data":"62e19dc79eeb4ebde645808d442474418df1612b437bf3323155ff3c31d3cbbc"}
Oct 30 00:50:25.456344 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:25.455959    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-be862c05-6b54db7858-9bx9r"
Oct 30 00:50:25.457700 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:25.457671    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-be862c05-c76879479-bfz2h" event={"ID":"8e75c787-c583-464d-b087-be6f84901779","Type":"ContainerStarted","Data":"172afc7d67157c22d3235b5902e5c2f5b7864be601b4ede37a8e7e1c36acfbaa"}
Oct 30 00:50:25.457915 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:25.457894    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-be862c05-c76879479-bfz2h"
Oct 30 00:50:25.498132 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:25.498084    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-be862c05-6b54db7858-9bx9r" podStartSLOduration=4.498071684 podStartE2EDuration="4.498071684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:25.494874099 +0000 UTC m=+1115.882550488" watchObservedRunningTime="2026-10-30 00:50:25.498071684 +0000 UTC m=+1115.885748072"
Oct 30 00:50:25.539515 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:25.539463    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-be862c05-c76879479-bfz2h" podStartSLOduration=4.539449028 podStartE2EDuration="4.539449028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:25.536837641 +0000 UTC m=+1115.924514031" watchObservedRunningTime="2026-10-30 00:50:25.539449028 +0000 UTC m=+1115.927125416"
Oct 30 00:50:25.936125 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:25.936096    4393 patch_prober.go:29] interesting pod/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 00:50:25.936319 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:25.936144    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8" podUID="eb8b407b-2bc7-4457-814b-31ffbdffa0c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 00:50:26.535217 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:50:26.535179    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:50:26.936419 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:26.936383    4393 patch_prober.go:29] interesting pod/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 00:50:26.936596 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:26.936452    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8" podUID="eb8b407b-2bc7-4457-814b-31ffbdffa0c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 00:50:27.935724 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:27.935688    4393 patch_prober.go:29] interesting pod/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 00:50:27.936185 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:27.935738    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8" podUID="eb8b407b-2bc7-4457-814b-31ffbdffa0c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 00:50:28.936108 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:28.936074    4393 patch_prober.go:29] interesting pod/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body=
Oct 30 00:50:28.936576 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:28.936132    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8" podUID="eb8b407b-2bc7-4457-814b-31ffbdffa0c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Oct 30 00:50:29.080240 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:29.080211    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-751d6789-7fb4c76b79-l6pp8"
Oct 30 00:50:29.080671 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:29.080652    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-751d6789-7bd9589f56-8vx54"
Oct 30 00:50:29.925512 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:29.925482    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl"]
Oct 30 00:50:29.930399 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:29.930375    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl"
Oct 30 00:50:29.942064 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:29.941016    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"
Oct 30 00:50:29.942064 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:29.942008    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"
Oct 30 00:50:29.943447 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:29.943427    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"
Oct 30 00:50:29.948620 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:29.948594    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-751d6789-serving-cert\""
Oct 30 00:50:29.970938 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:29.970908    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl"]
Oct 30 00:50:30.106923 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:30.106876    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0adbdcd7-c810-4578-8d5a-0b38960ec64c-openshift-service-ca\") pod \"maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl\" (UID: \"0adbdcd7-c810-4578-8d5a-0b38960ec64c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl"
Oct 30 00:50:30.106923 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:30.106926    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0adbdcd7-c810-4578-8d5a-0b38960ec64c-maas-api-tls\") pod \"maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl\" (UID: \"0adbdcd7-c810-4578-8d5a-0b38960ec64c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl"
Oct 30 00:50:30.107156 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:30.107134    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhzrs\" (UniqueName: \"kubernetes.io/projected/0adbdcd7-c810-4578-8d5a-0b38960ec64c-kube-api-access-dhzrs\") pod \"maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl\" (UID: \"0adbdcd7-c810-4578-8d5a-0b38960ec64c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl"
Oct 30 00:50:30.107330 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:30.107318    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0adbdcd7-c810-4578-8d5a-0b38960ec64c-metrics-certs\") pod \"maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl\" (UID: \"0adbdcd7-c810-4578-8d5a-0b38960ec64c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl"
Oct 30 00:50:30.212857 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:30.211979    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0adbdcd7-c810-4578-8d5a-0b38960ec64c-openshift-service-ca\") pod \"maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl\" (UID: \"0adbdcd7-c810-4578-8d5a-0b38960ec64c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl"
Oct 30 00:50:30.212857 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:30.212032    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0adbdcd7-c810-4578-8d5a-0b38960ec64c-maas-api-tls\") pod \"maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl\" (UID: \"0adbdcd7-c810-4578-8d5a-0b38960ec64c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl"
Oct 30 00:50:30.212857 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:30.212141    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhzrs\" (UniqueName: \"kubernetes.io/projected/0adbdcd7-c810-4578-8d5a-0b38960ec64c-kube-api-access-dhzrs\") pod \"maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl\" (UID: \"0adbdcd7-c810-4578-8d5a-0b38960ec64c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl"
Oct 30 00:50:30.212857 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:30.212394    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0adbdcd7-c810-4578-8d5a-0b38960ec64c-metrics-certs\") pod \"maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl\" (UID: \"0adbdcd7-c810-4578-8d5a-0b38960ec64c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl"
Oct 30 00:50:30.238096 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:30.236908    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0adbdcd7-c810-4578-8d5a-0b38960ec64c-openshift-service-ca\") pod \"maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl\" (UID: \"0adbdcd7-c810-4578-8d5a-0b38960ec64c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl"
Oct 30 00:50:30.252806 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:30.245213    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0adbdcd7-c810-4578-8d5a-0b38960ec64c-maas-api-tls\") pod \"maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl\" (UID: \"0adbdcd7-c810-4578-8d5a-0b38960ec64c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl"
Oct 30 00:50:30.252806 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:30.246585    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0adbdcd7-c810-4578-8d5a-0b38960ec64c-metrics-certs\") pod \"maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl\" (UID: \"0adbdcd7-c810-4578-8d5a-0b38960ec64c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl"
Oct 30 00:50:30.289818 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:30.287429    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhzrs\" (UniqueName: \"kubernetes.io/projected/0adbdcd7-c810-4578-8d5a-0b38960ec64c-kube-api-access-dhzrs\") pod \"maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl\" (UID: \"0adbdcd7-c810-4578-8d5a-0b38960ec64c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl"
Oct 30 00:50:30.548808 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:30.548707    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl"
Oct 30 00:50:30.861111 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:30.861088    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl"]
Oct 30 00:50:30.861248 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:50:30.861182    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0adbdcd7_c810_4578_8d5a_0b38960ec64c.slice/crio-6b957b78132e6b7a02eb458e185fde0f68f014c98e14df8ba41fe4327eee24fd WatchSource:0}: Error finding container 6b957b78132e6b7a02eb458e185fde0f68f014c98e14df8ba41fe4327eee24fd: Status 404 returned error can't find the container with id 6b957b78132e6b7a02eb458e185fde0f68f014c98e14df8ba41fe4327eee24fd
Oct 30 00:50:31.525606 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:31.525503    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl" event={"ID":"0adbdcd7-c810-4578-8d5a-0b38960ec64c","Type":"ContainerStarted","Data":"9fe599907c24667d56ededa3e7e2520933a1771903e01db4cbc24170f2ac9c3c"}
Oct 30 00:50:31.525606 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:31.525565    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl" event={"ID":"0adbdcd7-c810-4578-8d5a-0b38960ec64c","Type":"ContainerStarted","Data":"6b957b78132e6b7a02eb458e185fde0f68f014c98e14df8ba41fe4327eee24fd"}
Oct 30 00:50:31.525606 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:31.525594    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl"
Oct 30 00:50:31.570196 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:31.570141    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl" podStartSLOduration=2.157020266 podStartE2EDuration="2.570127698s" totalImagesPullingTime="413.107432ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:31.567501467 +0000 UTC m=+1121.955177857" watchObservedRunningTime="2026-10-30 00:50:31.570127698 +0000 UTC m=+1121.957804087"
Oct 30 00:50:35.010983 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:35.010947    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-be862c05-dcf9564b8-hfszp"]
Oct 30 00:50:35.015632 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:35.015615    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-be862c05-dcf9564b8-hfszp"
Oct 30 00:50:35.031158 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:35.031125    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-be862c05-serving-cert\""
Oct 30 00:50:35.050699 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:35.050666    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-be862c05-dcf9564b8-hfszp"]
Oct 30 00:50:35.163248 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:35.163213    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97b0fb2d-43b7-492d-b655-49c834f30bf8-maas-api-tls\") pod \"maas-api-e2e-derive-be862c05-dcf9564b8-hfszp\" (UID: \"97b0fb2d-43b7-492d-b655-49c834f30bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-be862c05-dcf9564b8-hfszp"
Oct 30 00:50:35.163445 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:35.163338    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfqdq\" (UniqueName: \"kubernetes.io/projected/97b0fb2d-43b7-492d-b655-49c834f30bf8-kube-api-access-wfqdq\") pod \"maas-api-e2e-derive-be862c05-dcf9564b8-hfszp\" (UID: \"97b0fb2d-43b7-492d-b655-49c834f30bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-be862c05-dcf9564b8-hfszp"
Oct 30 00:50:35.163445 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:35.163397    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97b0fb2d-43b7-492d-b655-49c834f30bf8-openshift-service-ca\") pod \"maas-api-e2e-derive-be862c05-dcf9564b8-hfszp\" (UID: \"97b0fb2d-43b7-492d-b655-49c834f30bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-be862c05-dcf9564b8-hfszp"
Oct 30 00:50:35.163577 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:35.163557    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97b0fb2d-43b7-492d-b655-49c834f30bf8-metrics-certs\") pod \"maas-api-e2e-derive-be862c05-dcf9564b8-hfszp\" (UID: \"97b0fb2d-43b7-492d-b655-49c834f30bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-be862c05-dcf9564b8-hfszp"
Oct 30 00:50:35.266364 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:35.266269    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfqdq\" (UniqueName: \"kubernetes.io/projected/97b0fb2d-43b7-492d-b655-49c834f30bf8-kube-api-access-wfqdq\") pod \"maas-api-e2e-derive-be862c05-dcf9564b8-hfszp\" (UID: \"97b0fb2d-43b7-492d-b655-49c834f30bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-be862c05-dcf9564b8-hfszp"
Oct 30 00:50:35.273982 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:35.273935    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97b0fb2d-43b7-492d-b655-49c834f30bf8-openshift-service-ca\") pod \"maas-api-e2e-derive-be862c05-dcf9564b8-hfszp\" (UID: \"97b0fb2d-43b7-492d-b655-49c834f30bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-be862c05-dcf9564b8-hfszp"
Oct 30 00:50:35.274741 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:35.274702    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97b0fb2d-43b7-492d-b655-49c834f30bf8-metrics-certs\") pod \"maas-api-e2e-derive-be862c05-dcf9564b8-hfszp\" (UID: \"97b0fb2d-43b7-492d-b655-49c834f30bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-be862c05-dcf9564b8-hfszp"
Oct 30 00:50:35.279823 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:35.275107    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97b0fb2d-43b7-492d-b655-49c834f30bf8-maas-api-tls\") pod \"maas-api-e2e-derive-be862c05-dcf9564b8-hfszp\" (UID: \"97b0fb2d-43b7-492d-b655-49c834f30bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-be862c05-dcf9564b8-hfszp"
Oct 30 00:50:35.280376 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:50:35.280344    4393 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-be862c05-serving-cert: secret "maas-api-e2e-derive-be862c05-serving-cert" not found
Oct 30 00:50:35.301954 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:50:35.298989    4393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97b0fb2d-43b7-492d-b655-49c834f30bf8-maas-api-tls podName:97b0fb2d-43b7-492d-b655-49c834f30bf8 nodeName:}" failed. No retries permitted until 2026-10-30 00:50:35.798955636 +0000 UTC m=+1126.186632017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/97b0fb2d-43b7-492d-b655-49c834f30bf8-maas-api-tls") pod "maas-api-e2e-derive-be862c05-dcf9564b8-hfszp" (UID: "97b0fb2d-43b7-492d-b655-49c834f30bf8") : secret "maas-api-e2e-derive-be862c05-serving-cert" not found
Oct 30 00:50:35.301954 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:35.284491    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97b0fb2d-43b7-492d-b655-49c834f30bf8-openshift-service-ca\") pod \"maas-api-e2e-derive-be862c05-dcf9564b8-hfszp\" (UID: \"97b0fb2d-43b7-492d-b655-49c834f30bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-be862c05-dcf9564b8-hfszp"
Oct 30 00:50:35.303894 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:35.303866    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfqdq\" (UniqueName: \"kubernetes.io/projected/97b0fb2d-43b7-492d-b655-49c834f30bf8-kube-api-access-wfqdq\") pod \"maas-api-e2e-derive-be862c05-dcf9564b8-hfszp\" (UID: \"97b0fb2d-43b7-492d-b655-49c834f30bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-be862c05-dcf9564b8-hfszp"
Oct 30 00:50:35.305255 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:35.305221    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97b0fb2d-43b7-492d-b655-49c834f30bf8-metrics-certs\") pod \"maas-api-e2e-derive-be862c05-dcf9564b8-hfszp\" (UID: \"97b0fb2d-43b7-492d-b655-49c834f30bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-be862c05-dcf9564b8-hfszp"
Oct 30 00:50:35.892000 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:35.891959    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97b0fb2d-43b7-492d-b655-49c834f30bf8-maas-api-tls\") pod \"maas-api-e2e-derive-be862c05-dcf9564b8-hfszp\" (UID: \"97b0fb2d-43b7-492d-b655-49c834f30bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-be862c05-dcf9564b8-hfszp"
Oct 30 00:50:35.903725 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:35.903695    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97b0fb2d-43b7-492d-b655-49c834f30bf8-maas-api-tls\") pod \"maas-api-e2e-derive-be862c05-dcf9564b8-hfszp\" (UID: \"97b0fb2d-43b7-492d-b655-49c834f30bf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-be862c05-dcf9564b8-hfszp"
Oct 30 00:50:35.930869 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:35.930840    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-be862c05-dcf9564b8-hfszp"
Oct 30 00:50:36.331567 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:36.331535    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-be862c05-dcf9564b8-hfszp"]
Oct 30 00:50:36.332071 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:50:36.332043    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b0fb2d_43b7_492d_b655_49c834f30bf8.slice/crio-4f73483afe595659c27bf93cfb79611e31ef173e2dfba42de61f6a6fc917fc53 WatchSource:0}: Error finding container 4f73483afe595659c27bf93cfb79611e31ef173e2dfba42de61f6a6fc917fc53: Status 404 returned error can't find the container with id 4f73483afe595659c27bf93cfb79611e31ef173e2dfba42de61f6a6fc917fc53
Oct 30 00:50:36.475531 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:36.475493    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-be862c05-c76879479-bfz2h"
Oct 30 00:50:36.481175 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:36.481144    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-be862c05-6b54db7858-9bx9r"
Oct 30 00:50:36.602113 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:36.601971    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-be862c05-dcf9564b8-hfszp" event={"ID":"97b0fb2d-43b7-492d-b655-49c834f30bf8","Type":"ContainerStarted","Data":"4f73483afe595659c27bf93cfb79611e31ef173e2dfba42de61f6a6fc917fc53"}
Oct 30 00:50:37.538316 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:37.538282    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl"
Oct 30 00:50:37.616052 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:37.616001    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-be862c05-dcf9564b8-hfszp" event={"ID":"97b0fb2d-43b7-492d-b655-49c834f30bf8","Type":"ContainerStarted","Data":"de8213555235948f048bf6c1a582973a2dc97f84dcd79afa3996e0e8f18e65b4"}
Oct 30 00:50:37.616213 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:37.616110    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-be862c05-dcf9564b8-hfszp"
Oct 30 00:50:37.659914 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:37.659865    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-be862c05-dcf9564b8-hfszp" podStartSLOduration=3.314461921 podStartE2EDuration="3.659851194s" totalImagesPullingTime="345.389273ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:37.657045188 +0000 UTC m=+1128.044721578" watchObservedRunningTime="2026-10-30 00:50:37.659851194 +0000 UTC m=+1128.047527582"
Oct 30 00:50:43.628317 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:43.628288    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-be862c05-dcf9564b8-hfszp"
Oct 30 00:50:55.581446 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:55.581414    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-11f521-6f8968b75-h6zh8"]
Oct 30 00:50:55.587852 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:55.587829    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-11f521-6f8968b75-h6zh8"
Oct 30 00:50:55.605119 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:55.605091    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-11f521-dockercfg-kd2zp\""
Oct 30 00:50:55.606888 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:55.606866    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-11f521-6f8968b75-h6zh8"]
Oct 30 00:50:55.611628 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:55.611598    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-11f521\""
Oct 30 00:50:55.676803 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:55.676745    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-11f521-7d8bfbf78-q7qwq"]
Oct 30 00:50:55.681327 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:55.681308    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-11f521-7d8bfbf78-q7qwq"
Oct 30 00:50:55.704191 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:55.704163    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-11f521-7d8bfbf78-q7qwq"]
Oct 30 00:50:55.789819 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:55.789588    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e044c11-fa02-43cd-af55-42b87c0b6789-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-11f521-6f8968b75-h6zh8\" (UID: \"1e044c11-fa02-43cd-af55-42b87c0b6789\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-11f521-6f8968b75-h6zh8"
Oct 30 00:50:55.790028 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:55.789899    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5rkd\" (UniqueName: \"kubernetes.io/projected/1e044c11-fa02-43cd-af55-42b87c0b6789-kube-api-access-k5rkd\") pod \"payload-pre-processing-e2e-shared-a-w3-11f521-6f8968b75-h6zh8\" (UID: \"1e044c11-fa02-43cd-af55-42b87c0b6789\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-11f521-6f8968b75-h6zh8"
Oct 30 00:50:55.930472 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:55.897938    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5rkd\" (UniqueName: \"kubernetes.io/projected/1e044c11-fa02-43cd-af55-42b87c0b6789-kube-api-access-k5rkd\") pod \"payload-pre-processing-e2e-shared-a-w3-11f521-6f8968b75-h6zh8\" (UID: \"1e044c11-fa02-43cd-af55-42b87c0b6789\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-11f521-6f8968b75-h6zh8"
Oct 30 00:50:55.930472 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:55.898047    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1ec248b-a567-4fb4-a415-ac1235608942-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-11f521-7d8bfbf78-q7qwq\" (UID: \"e1ec248b-a567-4fb4-a415-ac1235608942\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-11f521-7d8bfbf78-q7qwq"
Oct 30 00:50:55.930472 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:55.898268    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e044c11-fa02-43cd-af55-42b87c0b6789-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-11f521-6f8968b75-h6zh8\" (UID: \"1e044c11-fa02-43cd-af55-42b87c0b6789\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-11f521-6f8968b75-h6zh8"
Oct 30 00:50:55.930472 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:55.898321    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88s9p\" (UniqueName: \"kubernetes.io/projected/e1ec248b-a567-4fb4-a415-ac1235608942-kube-api-access-88s9p\") pod \"payload-processing-e2e-shared-a-w3-11f521-7d8bfbf78-q7qwq\" (UID: \"e1ec248b-a567-4fb4-a415-ac1235608942\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-11f521-7d8bfbf78-q7qwq"
Oct 30 00:50:55.930472 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:55.899738    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e044c11-fa02-43cd-af55-42b87c0b6789-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-11f521-6f8968b75-h6zh8\" (UID: \"1e044c11-fa02-43cd-af55-42b87c0b6789\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-11f521-6f8968b75-h6zh8"
Oct 30 00:50:55.990853 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:55.981633    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5rkd\" (UniqueName: \"kubernetes.io/projected/1e044c11-fa02-43cd-af55-42b87c0b6789-kube-api-access-k5rkd\") pod \"payload-pre-processing-e2e-shared-a-w3-11f521-6f8968b75-h6zh8\" (UID: \"1e044c11-fa02-43cd-af55-42b87c0b6789\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-11f521-6f8968b75-h6zh8"
Oct 30 00:50:56.007404 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:56.007364    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88s9p\" (UniqueName: \"kubernetes.io/projected/e1ec248b-a567-4fb4-a415-ac1235608942-kube-api-access-88s9p\") pod \"payload-processing-e2e-shared-a-w3-11f521-7d8bfbf78-q7qwq\" (UID: \"e1ec248b-a567-4fb4-a415-ac1235608942\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-11f521-7d8bfbf78-q7qwq"
Oct 30 00:50:56.008082 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:56.008056    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1ec248b-a567-4fb4-a415-ac1235608942-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-11f521-7d8bfbf78-q7qwq\" (UID: \"e1ec248b-a567-4fb4-a415-ac1235608942\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-11f521-7d8bfbf78-q7qwq"
Oct 30 00:50:56.016386 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:56.016343    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1ec248b-a567-4fb4-a415-ac1235608942-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-11f521-7d8bfbf78-q7qwq\" (UID: \"e1ec248b-a567-4fb4-a415-ac1235608942\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-11f521-7d8bfbf78-q7qwq"
Oct 30 00:50:56.064688 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:56.064042    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88s9p\" (UniqueName: \"kubernetes.io/projected/e1ec248b-a567-4fb4-a415-ac1235608942-kube-api-access-88s9p\") pod \"payload-processing-e2e-shared-a-w3-11f521-7d8bfbf78-q7qwq\" (UID: \"e1ec248b-a567-4fb4-a415-ac1235608942\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-11f521-7d8bfbf78-q7qwq"
Oct 30 00:50:56.235211 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:56.234915    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-11f521-6f8968b75-h6zh8"
Oct 30 00:50:56.305297 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:56.305222    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-11f521-7d8bfbf78-q7qwq"
Oct 30 00:50:56.513185 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:50:56.513096    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:50:56.555472 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:56.554249    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-11f521-6f8968b75-h6zh8"]
Oct 30 00:50:56.564254 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:50:56.564227    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e044c11_fa02_43cd_af55_42b87c0b6789.slice/crio-696379b0d6f5b89ba993c766a2baa08f110b79320cc658bf7089a77326cb3b6d WatchSource:0}: Error finding container 696379b0d6f5b89ba993c766a2baa08f110b79320cc658bf7089a77326cb3b6d: Status 404 returned error can't find the container with id 696379b0d6f5b89ba993c766a2baa08f110b79320cc658bf7089a77326cb3b6d
Oct 30 00:50:56.630889 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:50:56.630858    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1ec248b_a567_4fb4_a415_ac1235608942.slice/crio-a95d868386915aaf291dfad18f24ab589764c602dcdbf3e2d927ae07a400dfef WatchSource:0}: Error finding container a95d868386915aaf291dfad18f24ab589764c602dcdbf3e2d927ae07a400dfef: Status 404 returned error can't find the container with id a95d868386915aaf291dfad18f24ab589764c602dcdbf3e2d927ae07a400dfef
Oct 30 00:50:56.632170 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:56.632121    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-11f521-7d8bfbf78-q7qwq"]
Oct 30 00:50:57.079470 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:57.079430    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-11f521-7d8bfbf78-q7qwq" event={"ID":"e1ec248b-a567-4fb4-a415-ac1235608942","Type":"ContainerStarted","Data":"4783c92d9b0807d6838d68e0b0459c912aed4962ac893390169847137194cb31"}
Oct 30 00:50:57.079671 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:57.079494    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-11f521-7d8bfbf78-q7qwq" event={"ID":"e1ec248b-a567-4fb4-a415-ac1235608942","Type":"ContainerStarted","Data":"a95d868386915aaf291dfad18f24ab589764c602dcdbf3e2d927ae07a400dfef"}
Oct 30 00:50:57.079671 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:57.079519    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-11f521-7d8bfbf78-q7qwq"
Oct 30 00:50:57.081421 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:57.081392    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-11f521-6f8968b75-h6zh8" event={"ID":"1e044c11-fa02-43cd-af55-42b87c0b6789","Type":"ContainerStarted","Data":"5a6a1ca86230a4ab7704a1d3382dfcce5e519bf6d8f66d0cee9da0085acb67be"}
Oct 30 00:50:57.081582 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:57.081441    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-11f521-6f8968b75-h6zh8" event={"ID":"1e044c11-fa02-43cd-af55-42b87c0b6789","Type":"ContainerStarted","Data":"696379b0d6f5b89ba993c766a2baa08f110b79320cc658bf7089a77326cb3b6d"}
Oct 30 00:50:57.081582 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:57.081473    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-11f521-6f8968b75-h6zh8"
Oct 30 00:50:57.118468 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:57.118422    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-11f521-7d8bfbf78-q7qwq" podStartSLOduration=2.118411492 podStartE2EDuration="2.118411492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:57.117832399 +0000 UTC m=+1147.505508791" watchObservedRunningTime="2026-10-30 00:50:57.118411492 +0000 UTC m=+1147.506087897"
Oct 30 00:50:57.157147 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:50:57.157098    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-11f521-6f8968b75-h6zh8" podStartSLOduration=2.157084235 podStartE2EDuration="2.157084235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:50:57.156036556 +0000 UTC m=+1147.543712947" watchObservedRunningTime="2026-10-30 00:50:57.157084235 +0000 UTC m=+1147.544760624"
Oct 30 00:51:01.695881 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:01.695848    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-751d6789-7bd9589f56-8vx54"]
Oct 30 00:51:01.729374 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:01.729347    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-751d6789-7fb4c76b79-l6pp8"]
Oct 30 00:51:06.007118 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:06.007083    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl"]
Oct 30 00:51:06.007597 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:06.007296    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl" podUID="0adbdcd7-c810-4578-8d5a-0b38960ec64c" containerName="maas-api" containerID="cri-o://9fe599907c24667d56ededa3e7e2520933a1771903e01db4cbc24170f2ac9c3c" gracePeriod=30
Oct 30 00:51:06.202500 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:06.202469    4393 generic.go:356] "Generic (PLEG): container finished" podID="0adbdcd7-c810-4578-8d5a-0b38960ec64c" containerID="9fe599907c24667d56ededa3e7e2520933a1771903e01db4cbc24170f2ac9c3c" exitCode=0
Oct 30 00:51:06.202657 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:06.202531    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl" event={"ID":"0adbdcd7-c810-4578-8d5a-0b38960ec64c","Type":"ContainerDied","Data":"9fe599907c24667d56ededa3e7e2520933a1771903e01db4cbc24170f2ac9c3c"}
Oct 30 00:51:06.527306 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:06.527270    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl"
Oct 30 00:51:06.652807 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:06.639944    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0adbdcd7-c810-4578-8d5a-0b38960ec64c-metrics-certs" pod "0adbdcd7-c810-4578-8d5a-0b38960ec64c" (UID: "0adbdcd7-c810-4578-8d5a-0b38960ec64c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:51:06.711858 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:06.708863    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-751d6789-7bd9589f56-8vx54" podUID="f5f99387-2f87-423c-89ca-1f7676f8c4c6" containerName="payload-processing" containerID="cri-o://d607e65dee32cc7254eec77807128c933a0b885476670294fa63a3b80800ca8d" gracePeriod=25
Oct 30 00:51:06.750816 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:06.734457    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-751d6789-7fb4c76b79-l6pp8" podUID="8e8b8136-7f22-47fb-8c7c-bd11e4bb6281" containerName="payload-pre-processing" containerID="cri-o://2306f1b6df9ca352242ba431d190507c96557c46fb1937c19dd5a97d48381f96" gracePeriod=25
Oct 30 00:51:06.760912 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:06.759009    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0adbdcd7-c810-4578-8d5a-0b38960ec64c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0adbdcd7-c810-4578-8d5a-0b38960ec64c-metrics-certs\") pod \"0adbdcd7-c810-4578-8d5a-0b38960ec64c\" (UID: \"0adbdcd7-c810-4578-8d5a-0b38960ec64c\") "
Oct 30 00:51:06.760912 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:06.759101    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0adbdcd7-c810-4578-8d5a-0b38960ec64c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0adbdcd7-c810-4578-8d5a-0b38960ec64c-maas-api-tls\") pod \"0adbdcd7-c810-4578-8d5a-0b38960ec64c\" (UID: \"0adbdcd7-c810-4578-8d5a-0b38960ec64c\") "
Oct 30 00:51:06.760912 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:06.759202    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0adbdcd7-c810-4578-8d5a-0b38960ec64c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0adbdcd7-c810-4578-8d5a-0b38960ec64c-openshift-service-ca\") pod \"0adbdcd7-c810-4578-8d5a-0b38960ec64c\" (UID: \"0adbdcd7-c810-4578-8d5a-0b38960ec64c\") "
Oct 30 00:51:06.760912 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:06.759252    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0adbdcd7-c810-4578-8d5a-0b38960ec64c-kube-api-access-dhzrs\" (UniqueName: \"kubernetes.io/projected/0adbdcd7-c810-4578-8d5a-0b38960ec64c-kube-api-access-dhzrs\") pod \"0adbdcd7-c810-4578-8d5a-0b38960ec64c\" (UID: \"0adbdcd7-c810-4578-8d5a-0b38960ec64c\") "
Oct 30 00:51:06.761604 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:06.761536    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0adbdcd7-c810-4578-8d5a-0b38960ec64c-openshift-service-ca" pod "0adbdcd7-c810-4578-8d5a-0b38960ec64c" (UID: "0adbdcd7-c810-4578-8d5a-0b38960ec64c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:51:06.784424 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:06.783967    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0adbdcd7-c810-4578-8d5a-0b38960ec64c-maas-api-tls" pod "0adbdcd7-c810-4578-8d5a-0b38960ec64c" (UID: "0adbdcd7-c810-4578-8d5a-0b38960ec64c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:51:06.847734 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:06.844989    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0adbdcd7-c810-4578-8d5a-0b38960ec64c-kube-api-access-dhzrs" pod "0adbdcd7-c810-4578-8d5a-0b38960ec64c" (UID: "0adbdcd7-c810-4578-8d5a-0b38960ec64c"). InnerVolumeSpecName "kube-api-access-dhzrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:51:06.861462 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:06.861418    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0adbdcd7-c810-4578-8d5a-0b38960ec64c-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:51:06.861462 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:06.861457    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0adbdcd7-c810-4578-8d5a-0b38960ec64c-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:51:06.861662 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:06.861475    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0adbdcd7-c810-4578-8d5a-0b38960ec64c-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:51:06.861662 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:06.861492    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhzrs\" (UniqueName: \"kubernetes.io/projected/0adbdcd7-c810-4578-8d5a-0b38960ec64c-kube-api-access-dhzrs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:51:07.186566 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:07.186539    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-751d6789-7fb4c76b79-l6pp8"
Oct 30 00:51:07.192832 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:07.192766    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e8b8136-7f22-47fb-8c7c-bd11e4bb6281-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e8b8136-7f22-47fb-8c7c-bd11e4bb6281-plugins-config-volume\") pod \"8e8b8136-7f22-47fb-8c7c-bd11e4bb6281\" (UID: \"8e8b8136-7f22-47fb-8c7c-bd11e4bb6281\") "
Oct 30 00:51:07.193026 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:07.193010    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e8b8136-7f22-47fb-8c7c-bd11e4bb6281-kube-api-access-ngkv8\" (UniqueName: \"kubernetes.io/projected/8e8b8136-7f22-47fb-8c7c-bd11e4bb6281-kube-api-access-ngkv8\") pod \"8e8b8136-7f22-47fb-8c7c-bd11e4bb6281\" (UID: \"8e8b8136-7f22-47fb-8c7c-bd11e4bb6281\") "
Oct 30 00:51:07.193127 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:07.193070    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e8b8136-7f22-47fb-8c7c-bd11e4bb6281-plugins-config-volume" pod "8e8b8136-7f22-47fb-8c7c-bd11e4bb6281" (UID: "8e8b8136-7f22-47fb-8c7c-bd11e4bb6281"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:51:07.193632 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:07.193609    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e8b8136-7f22-47fb-8c7c-bd11e4bb6281-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:51:07.195941 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:07.195771    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e8b8136-7f22-47fb-8c7c-bd11e4bb6281-kube-api-access-ngkv8" pod "8e8b8136-7f22-47fb-8c7c-bd11e4bb6281" (UID: "8e8b8136-7f22-47fb-8c7c-bd11e4bb6281"). InnerVolumeSpecName "kube-api-access-ngkv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:51:07.222185 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:07.222155    4393 generic.go:356] "Generic (PLEG): container finished" podID="f5f99387-2f87-423c-89ca-1f7676f8c4c6" containerID="d607e65dee32cc7254eec77807128c933a0b885476670294fa63a3b80800ca8d" exitCode=0
Oct 30 00:51:07.222320 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:07.222233    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-751d6789-7bd9589f56-8vx54" event={"ID":"f5f99387-2f87-423c-89ca-1f7676f8c4c6","Type":"ContainerDied","Data":"d607e65dee32cc7254eec77807128c933a0b885476670294fa63a3b80800ca8d"}
Oct 30 00:51:07.228829 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:07.228166    4393 generic.go:356] "Generic (PLEG): container finished" podID="8e8b8136-7f22-47fb-8c7c-bd11e4bb6281" containerID="2306f1b6df9ca352242ba431d190507c96557c46fb1937c19dd5a97d48381f96" exitCode=0
Oct 30 00:51:07.228829 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:07.228265    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-751d6789-7fb4c76b79-l6pp8" event={"ID":"8e8b8136-7f22-47fb-8c7c-bd11e4bb6281","Type":"ContainerDied","Data":"2306f1b6df9ca352242ba431d190507c96557c46fb1937c19dd5a97d48381f96"}
Oct 30 00:51:07.228829 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:07.228315    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-751d6789-7fb4c76b79-l6pp8" event={"ID":"8e8b8136-7f22-47fb-8c7c-bd11e4bb6281","Type":"ContainerDied","Data":"06e7307ae1eb42e286747a14a5847d8e667fc0d513dbdc544aff61acedc7a33a"}
Oct 30 00:51:07.228829 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:07.228347    4393 scope.go:122] "RemoveContainer" containerID="2306f1b6df9ca352242ba431d190507c96557c46fb1937c19dd5a97d48381f96"
Oct 30 00:51:07.228829 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:07.228527    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-751d6789-7fb4c76b79-l6pp8"
Oct 30 00:51:07.243571 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:07.243469    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl" event={"ID":"0adbdcd7-c810-4578-8d5a-0b38960ec64c","Type":"ContainerDied","Data":"6b957b78132e6b7a02eb458e185fde0f68f014c98e14df8ba41fe4327eee24fd"}
Oct 30 00:51:07.243571 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:07.243570    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl"
Oct 30 00:51:07.253906 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:07.253868    4393 scope.go:122] "RemoveContainer" containerID="2306f1b6df9ca352242ba431d190507c96557c46fb1937c19dd5a97d48381f96"
Oct 30 00:51:07.254385 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:51:07.254235    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2306f1b6df9ca352242ba431d190507c96557c46fb1937c19dd5a97d48381f96\": container with ID starting with 2306f1b6df9ca352242ba431d190507c96557c46fb1937c19dd5a97d48381f96 not found: ID does not exist" containerID="2306f1b6df9ca352242ba431d190507c96557c46fb1937c19dd5a97d48381f96"
Oct 30 00:51:07.254385 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:07.254267    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2306f1b6df9ca352242ba431d190507c96557c46fb1937c19dd5a97d48381f96"} err="failed to get container status \"2306f1b6df9ca352242ba431d190507c96557c46fb1937c19dd5a97d48381f96\": rpc error: code = NotFound desc = could not find container \"2306f1b6df9ca352242ba431d190507c96557c46fb1937c19dd5a97d48381f96\": container with ID starting with 2306f1b6df9ca352242ba431d190507c96557c46fb1937c19dd5a97d48381f96 not found: ID does not exist"
Oct 30 00:51:07.254385 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:07.254294    4393 scope.go:122] "RemoveContainer" containerID="9fe599907c24667d56ededa3e7e2520933a1771903e01db4cbc24170f2ac9c3c"
Oct 30 00:51:07.295050 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:07.294952    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngkv8\" (UniqueName: \"kubernetes.io/projected/8e8b8136-7f22-47fb-8c7c-bd11e4bb6281-kube-api-access-ngkv8\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:51:07.303631 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:07.303561    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-751d6789-7fb4c76b79-l6pp8"]
Oct 30 00:51:07.308961 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:07.308936    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-751d6789-7fb4c76b79-l6pp8"]
Oct 30 00:51:07.322961 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:07.322937    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-751d6789-7bd9589f56-8vx54"
Oct 30 00:51:07.336464 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:07.336403    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl"]
Oct 30 00:51:07.340393 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:07.340374    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-751d6789-7cf95bf689-b6rsl"]
Oct 30 00:51:07.395635 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:07.395605    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5f99387-2f87-423c-89ca-1f7676f8c4c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5f99387-2f87-423c-89ca-1f7676f8c4c6-plugins-config-volume\") pod \"f5f99387-2f87-423c-89ca-1f7676f8c4c6\" (UID: \"f5f99387-2f87-423c-89ca-1f7676f8c4c6\") "
Oct 30 00:51:07.396022 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:07.395999    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5f99387-2f87-423c-89ca-1f7676f8c4c6-kube-api-access-q6bdr\" (UniqueName: \"kubernetes.io/projected/f5f99387-2f87-423c-89ca-1f7676f8c4c6-kube-api-access-q6bdr\") pod \"f5f99387-2f87-423c-89ca-1f7676f8c4c6\" (UID: \"f5f99387-2f87-423c-89ca-1f7676f8c4c6\") "
Oct 30 00:51:07.396495 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:07.396113    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5f99387-2f87-423c-89ca-1f7676f8c4c6-plugins-config-volume" pod "f5f99387-2f87-423c-89ca-1f7676f8c4c6" (UID: "f5f99387-2f87-423c-89ca-1f7676f8c4c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:51:07.396580 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:07.396547    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5f99387-2f87-423c-89ca-1f7676f8c4c6-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:51:07.399126 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:07.399099    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5f99387-2f87-423c-89ca-1f7676f8c4c6-kube-api-access-q6bdr" pod "f5f99387-2f87-423c-89ca-1f7676f8c4c6" (UID: "f5f99387-2f87-423c-89ca-1f7676f8c4c6"). InnerVolumeSpecName "kube-api-access-q6bdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:51:07.498038 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:07.498006    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6bdr\" (UniqueName: \"kubernetes.io/projected/f5f99387-2f87-423c-89ca-1f7676f8c4c6-kube-api-access-q6bdr\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:51:07.726229 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:07.726198    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0adbdcd7-c810-4578-8d5a-0b38960ec64c" path="/var/lib/kubelet/pods/0adbdcd7-c810-4578-8d5a-0b38960ec64c/volumes"
Oct 30 00:51:07.726622 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:07.726598    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e8b8136-7f22-47fb-8c7c-bd11e4bb6281" path="/var/lib/kubelet/pods/8e8b8136-7f22-47fb-8c7c-bd11e4bb6281/volumes"
Oct 30 00:51:08.121460 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:08.121428    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-11f521-7d8bfbf78-q7qwq"
Oct 30 00:51:08.121842 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:08.121818    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-11f521-6f8968b75-h6zh8"
Oct 30 00:51:08.256891 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:08.256846    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-751d6789-7bd9589f56-8vx54" event={"ID":"f5f99387-2f87-423c-89ca-1f7676f8c4c6","Type":"ContainerDied","Data":"d15289e9e0a9f43fbd061e5ba0c44b764404867bbf643c12c4d4e6601f2a045c"}
Oct 30 00:51:08.257364 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:08.256874    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-751d6789-7bd9589f56-8vx54"
Oct 30 00:51:08.257364 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:08.256965    4393 scope.go:122] "RemoveContainer" containerID="d607e65dee32cc7254eec77807128c933a0b885476670294fa63a3b80800ca8d"
Oct 30 00:51:08.311679 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:08.311641    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-751d6789-7bd9589f56-8vx54"]
Oct 30 00:51:08.328395 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:08.328312    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-751d6789-7bd9589f56-8vx54"]
Oct 30 00:51:09.727917 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:09.727886    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5f99387-2f87-423c-89ca-1f7676f8c4c6" path="/var/lib/kubelet/pods/f5f99387-2f87-423c-89ca-1f7676f8c4c6/volumes"
Oct 30 00:51:10.419019 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:10.418986    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52"]
Oct 30 00:51:10.420014 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:10.419989    4393 cpu_manager.go:414] "containerMap: removing container" podUID="f5f99387-2f87-423c-89ca-1f7676f8c4c6" containerName="payload-processing"
Oct 30 00:51:10.420014 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:10.420013    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5f99387-2f87-423c-89ca-1f7676f8c4c6" containerName="payload-processing"
Oct 30 00:51:10.420213 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:10.420075    4393 cpu_manager.go:414] "containerMap: removing container" podUID="0adbdcd7-c810-4578-8d5a-0b38960ec64c" containerName="maas-api"
Oct 30 00:51:10.420213 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:10.420084    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0adbdcd7-c810-4578-8d5a-0b38960ec64c" containerName="maas-api"
Oct 30 00:51:10.420213 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:10.420103    4393 cpu_manager.go:414] "containerMap: removing container" podUID="8e8b8136-7f22-47fb-8c7c-bd11e4bb6281" containerName="payload-pre-processing"
Oct 30 00:51:10.420213 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:10.420111    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e8b8136-7f22-47fb-8c7c-bd11e4bb6281" containerName="payload-pre-processing"
Oct 30 00:51:10.420424 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:10.420284    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="0adbdcd7-c810-4578-8d5a-0b38960ec64c" containerName="maas-api"
Oct 30 00:51:10.420424 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:10.420301    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e8b8136-7f22-47fb-8c7c-bd11e4bb6281" containerName="payload-pre-processing"
Oct 30 00:51:10.420424 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:10.420320    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5f99387-2f87-423c-89ca-1f7676f8c4c6" containerName="payload-processing"
Oct 30 00:51:10.426102 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:10.426079    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52"
Oct 30 00:51:10.444230 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:10.443910    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-11f521-serving-cert\""
Oct 30 00:51:10.451537 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:10.451505    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52"]
Oct 30 00:51:10.585500 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:10.585461    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a88ea59f-24dc-42df-8bd5-ffffdb5f36f2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52\" (UID: \"a88ea59f-24dc-42df-8bd5-ffffdb5f36f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52"
Oct 30 00:51:10.585695 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:10.585536    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a88ea59f-24dc-42df-8bd5-ffffdb5f36f2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52\" (UID: \"a88ea59f-24dc-42df-8bd5-ffffdb5f36f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52"
Oct 30 00:51:10.585695 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:10.585603    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a88ea59f-24dc-42df-8bd5-ffffdb5f36f2-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52\" (UID: \"a88ea59f-24dc-42df-8bd5-ffffdb5f36f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52"
Oct 30 00:51:10.585880 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:10.585724    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmfvd\" (UniqueName: \"kubernetes.io/projected/a88ea59f-24dc-42df-8bd5-ffffdb5f36f2-kube-api-access-hmfvd\") pod \"maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52\" (UID: \"a88ea59f-24dc-42df-8bd5-ffffdb5f36f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52"
Oct 30 00:51:10.705857 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:10.691286    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a88ea59f-24dc-42df-8bd5-ffffdb5f36f2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52\" (UID: \"a88ea59f-24dc-42df-8bd5-ffffdb5f36f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52"
Oct 30 00:51:10.705857 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:10.691347    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a88ea59f-24dc-42df-8bd5-ffffdb5f36f2-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52\" (UID: \"a88ea59f-24dc-42df-8bd5-ffffdb5f36f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52"
Oct 30 00:51:10.705857 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:10.691499    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmfvd\" (UniqueName: \"kubernetes.io/projected/a88ea59f-24dc-42df-8bd5-ffffdb5f36f2-kube-api-access-hmfvd\") pod \"maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52\" (UID: \"a88ea59f-24dc-42df-8bd5-ffffdb5f36f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52"
Oct 30 00:51:10.705857 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:10.691746    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a88ea59f-24dc-42df-8bd5-ffffdb5f36f2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52\" (UID: \"a88ea59f-24dc-42df-8bd5-ffffdb5f36f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52"
Oct 30 00:51:10.705857 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:10.692568    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a88ea59f-24dc-42df-8bd5-ffffdb5f36f2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52\" (UID: \"a88ea59f-24dc-42df-8bd5-ffffdb5f36f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52"
Oct 30 00:51:10.709318 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:10.709290    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a88ea59f-24dc-42df-8bd5-ffffdb5f36f2-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52\" (UID: \"a88ea59f-24dc-42df-8bd5-ffffdb5f36f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52"
Oct 30 00:51:10.735877 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:10.735845    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a88ea59f-24dc-42df-8bd5-ffffdb5f36f2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52\" (UID: \"a88ea59f-24dc-42df-8bd5-ffffdb5f36f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52"
Oct 30 00:51:10.736926 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:10.736857    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmfvd\" (UniqueName: \"kubernetes.io/projected/a88ea59f-24dc-42df-8bd5-ffffdb5f36f2-kube-api-access-hmfvd\") pod \"maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52\" (UID: \"a88ea59f-24dc-42df-8bd5-ffffdb5f36f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52"
Oct 30 00:51:10.743700 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:10.743046    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52"
Oct 30 00:51:11.035922 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:11.035890    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-be862c05-6b54db7858-9bx9r"]
Oct 30 00:51:11.072070 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:11.072032    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-be862c05-c76879479-bfz2h"]
Oct 30 00:51:11.240052 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:11.240024    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52"]
Oct 30 00:51:11.240300 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:51:11.240271    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda88ea59f_24dc_42df_8bd5_ffffdb5f36f2.slice/crio-30227892e2becfbbf6ea66bdd84b4a77f8afaf40dd8e6959e52eae6efd176ae4 WatchSource:0}: Error finding container 30227892e2becfbbf6ea66bdd84b4a77f8afaf40dd8e6959e52eae6efd176ae4: Status 404 returned error can't find the container with id 30227892e2becfbbf6ea66bdd84b4a77f8afaf40dd8e6959e52eae6efd176ae4
Oct 30 00:51:11.285852 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:11.285821    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52" event={"ID":"a88ea59f-24dc-42df-8bd5-ffffdb5f36f2","Type":"ContainerStarted","Data":"30227892e2becfbbf6ea66bdd84b4a77f8afaf40dd8e6959e52eae6efd176ae4"}
Oct 30 00:51:12.086977 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:12.086867    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-be862c05-dcf9564b8-hfszp"]
Oct 30 00:51:12.087470 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:12.087162    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-be862c05-dcf9564b8-hfszp" podUID="97b0fb2d-43b7-492d-b655-49c834f30bf8" containerName="maas-api" containerID="cri-o://de8213555235948f048bf6c1a582973a2dc97f84dcd79afa3996e0e8f18e65b4" gracePeriod=30
Oct 30 00:51:12.302160 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:12.302124    4393 generic.go:356] "Generic (PLEG): container finished" podID="97b0fb2d-43b7-492d-b655-49c834f30bf8" containerID="de8213555235948f048bf6c1a582973a2dc97f84dcd79afa3996e0e8f18e65b4" exitCode=0
Oct 30 00:51:12.302405 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:12.302223    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-be862c05-dcf9564b8-hfszp" event={"ID":"97b0fb2d-43b7-492d-b655-49c834f30bf8","Type":"ContainerDied","Data":"de8213555235948f048bf6c1a582973a2dc97f84dcd79afa3996e0e8f18e65b4"}
Oct 30 00:51:12.305008 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:12.304203    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52" event={"ID":"a88ea59f-24dc-42df-8bd5-ffffdb5f36f2","Type":"ContainerStarted","Data":"38d7860401974c4370a1853fd2c2a72b9470a34d61a57e7097a771479275b2bc"}
Oct 30 00:51:12.305008 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:12.304447    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52"
Oct 30 00:51:12.470842 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:12.449195    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52" podStartSLOduration=2.032011281 podStartE2EDuration="2.449174444s" totalImagesPullingTime="417.163163ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:12.388897596 +0000 UTC m=+1162.776573985" watchObservedRunningTime="2026-10-30 00:51:12.449174444 +0000 UTC m=+1162.836850831"
Oct 30 00:51:12.551103 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:12.551073    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-be862c05-dcf9564b8-hfszp"
Oct 30 00:51:12.701944 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:12.701852    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97b0fb2d-43b7-492d-b655-49c834f30bf8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/97b0fb2d-43b7-492d-b655-49c834f30bf8-metrics-certs\") pod \"97b0fb2d-43b7-492d-b655-49c834f30bf8\" (UID: \"97b0fb2d-43b7-492d-b655-49c834f30bf8\") "
Oct 30 00:51:12.702219 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:12.702199    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97b0fb2d-43b7-492d-b655-49c834f30bf8-kube-api-access-wfqdq\" (UniqueName: \"kubernetes.io/projected/97b0fb2d-43b7-492d-b655-49c834f30bf8-kube-api-access-wfqdq\") pod \"97b0fb2d-43b7-492d-b655-49c834f30bf8\" (UID: \"97b0fb2d-43b7-492d-b655-49c834f30bf8\") "
Oct 30 00:51:12.702437 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:12.702420    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97b0fb2d-43b7-492d-b655-49c834f30bf8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97b0fb2d-43b7-492d-b655-49c834f30bf8-maas-api-tls\") pod \"97b0fb2d-43b7-492d-b655-49c834f30bf8\" (UID: \"97b0fb2d-43b7-492d-b655-49c834f30bf8\") "
Oct 30 00:51:12.702598 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:12.702581    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97b0fb2d-43b7-492d-b655-49c834f30bf8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97b0fb2d-43b7-492d-b655-49c834f30bf8-openshift-service-ca\") pod \"97b0fb2d-43b7-492d-b655-49c834f30bf8\" (UID: \"97b0fb2d-43b7-492d-b655-49c834f30bf8\") "
Oct 30 00:51:12.703826 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:12.703799    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97b0fb2d-43b7-492d-b655-49c834f30bf8-openshift-service-ca" pod "97b0fb2d-43b7-492d-b655-49c834f30bf8" (UID: "97b0fb2d-43b7-492d-b655-49c834f30bf8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:51:12.791208 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:12.724283    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97b0fb2d-43b7-492d-b655-49c834f30bf8-metrics-certs" pod "97b0fb2d-43b7-492d-b655-49c834f30bf8" (UID: "97b0fb2d-43b7-492d-b655-49c834f30bf8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:51:12.791208 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:12.744298    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97b0fb2d-43b7-492d-b655-49c834f30bf8-maas-api-tls" pod "97b0fb2d-43b7-492d-b655-49c834f30bf8" (UID: "97b0fb2d-43b7-492d-b655-49c834f30bf8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:51:12.791208 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:12.747495    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97b0fb2d-43b7-492d-b655-49c834f30bf8-kube-api-access-wfqdq" pod "97b0fb2d-43b7-492d-b655-49c834f30bf8" (UID: "97b0fb2d-43b7-492d-b655-49c834f30bf8"). InnerVolumeSpecName "kube-api-access-wfqdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:51:12.867806 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:12.860319    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfqdq\" (UniqueName: \"kubernetes.io/projected/97b0fb2d-43b7-492d-b655-49c834f30bf8-kube-api-access-wfqdq\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:51:12.867806 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:12.860353    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97b0fb2d-43b7-492d-b655-49c834f30bf8-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:51:12.867806 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:12.860368    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97b0fb2d-43b7-492d-b655-49c834f30bf8-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:51:12.867806 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:12.860383    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97b0fb2d-43b7-492d-b655-49c834f30bf8-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:51:13.469828 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:13.464928    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-be862c05-dcf9564b8-hfszp"
Oct 30 00:51:13.476218 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:13.476178    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-be862c05-dcf9564b8-hfszp" event={"ID":"97b0fb2d-43b7-492d-b655-49c834f30bf8","Type":"ContainerDied","Data":"4f73483afe595659c27bf93cfb79611e31ef173e2dfba42de61f6a6fc917fc53"}
Oct 30 00:51:13.476465 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:13.476449    4393 scope.go:122] "RemoveContainer" containerID="de8213555235948f048bf6c1a582973a2dc97f84dcd79afa3996e0e8f18e65b4"
Oct 30 00:51:13.570832 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:13.567623    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-be862c05-dcf9564b8-hfszp"]
Oct 30 00:51:13.579802 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:13.576864    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-be862c05-dcf9564b8-hfszp"]
Oct 30 00:51:13.726985 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:13.726904    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97b0fb2d-43b7-492d-b655-49c834f30bf8" path="/var/lib/kubelet/pods/97b0fb2d-43b7-492d-b655-49c834f30bf8/volumes"
Oct 30 00:51:16.037137 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:16.037082    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-be862c05-6b54db7858-9bx9r" podUID="db72d8fa-fc86-4654-b097-7d6a920a4d72" containerName="payload-processing" containerID="cri-o://62e19dc79eeb4ebde645808d442474418df1612b437bf3323155ff3c31d3cbbc" gracePeriod=25
Oct 30 00:51:16.072615 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:16.072575    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-be862c05-c76879479-bfz2h" podUID="8e75c787-c583-464d-b087-be6f84901779" containerName="payload-pre-processing" containerID="cri-o://172afc7d67157c22d3235b5902e5c2f5b7864be601b4ede37a8e7e1c36acfbaa" gracePeriod=25
Oct 30 00:51:16.379796 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:16.379761    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-be862c05-6b54db7858-9bx9r"
Oct 30 00:51:16.406024 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:16.405977    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-be862c05-c76879479-bfz2h"
Oct 30 00:51:16.465224 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:16.465181    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db72d8fa-fc86-4654-b097-7d6a920a4d72-kube-api-access-vqrlc\" (UniqueName: \"kubernetes.io/projected/db72d8fa-fc86-4654-b097-7d6a920a4d72-kube-api-access-vqrlc\") pod \"db72d8fa-fc86-4654-b097-7d6a920a4d72\" (UID: \"db72d8fa-fc86-4654-b097-7d6a920a4d72\") "
Oct 30 00:51:16.465423 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:16.465243    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db72d8fa-fc86-4654-b097-7d6a920a4d72-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db72d8fa-fc86-4654-b097-7d6a920a4d72-plugins-config-volume\") pod \"db72d8fa-fc86-4654-b097-7d6a920a4d72\" (UID: \"db72d8fa-fc86-4654-b097-7d6a920a4d72\") "
Oct 30 00:51:16.465423 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:16.465331    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e75c787-c583-464d-b087-be6f84901779-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e75c787-c583-464d-b087-be6f84901779-plugins-config-volume\") pod \"8e75c787-c583-464d-b087-be6f84901779\" (UID: \"8e75c787-c583-464d-b087-be6f84901779\") "
Oct 30 00:51:16.465663 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:16.465625    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e75c787-c583-464d-b087-be6f84901779-kube-api-access-mcqnf\" (UniqueName: \"kubernetes.io/projected/8e75c787-c583-464d-b087-be6f84901779-kube-api-access-mcqnf\") pod \"8e75c787-c583-464d-b087-be6f84901779\" (UID: \"8e75c787-c583-464d-b087-be6f84901779\") "
Oct 30 00:51:16.465752 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:16.465678    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db72d8fa-fc86-4654-b097-7d6a920a4d72-plugins-config-volume" pod "db72d8fa-fc86-4654-b097-7d6a920a4d72" (UID: "db72d8fa-fc86-4654-b097-7d6a920a4d72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:51:16.466159 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:16.466129    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e75c787-c583-464d-b087-be6f84901779-plugins-config-volume" pod "8e75c787-c583-464d-b087-be6f84901779" (UID: "8e75c787-c583-464d-b087-be6f84901779"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:51:16.466542 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:16.466386    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db72d8fa-fc86-4654-b097-7d6a920a4d72-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:51:16.466542 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:16.466428    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e75c787-c583-464d-b087-be6f84901779-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:51:16.468264 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:16.468238    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db72d8fa-fc86-4654-b097-7d6a920a4d72-kube-api-access-vqrlc" pod "db72d8fa-fc86-4654-b097-7d6a920a4d72" (UID: "db72d8fa-fc86-4654-b097-7d6a920a4d72"). InnerVolumeSpecName "kube-api-access-vqrlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:51:16.468360 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:16.468333    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e75c787-c583-464d-b087-be6f84901779-kube-api-access-mcqnf" pod "8e75c787-c583-464d-b087-be6f84901779" (UID: "8e75c787-c583-464d-b087-be6f84901779"). InnerVolumeSpecName "kube-api-access-mcqnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:51:16.493299 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:16.493266    4393 generic.go:356] "Generic (PLEG): container finished" podID="db72d8fa-fc86-4654-b097-7d6a920a4d72" containerID="62e19dc79eeb4ebde645808d442474418df1612b437bf3323155ff3c31d3cbbc" exitCode=0
Oct 30 00:51:16.493479 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:16.493315    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-be862c05-6b54db7858-9bx9r"
Oct 30 00:51:16.493479 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:16.493355    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-be862c05-6b54db7858-9bx9r" event={"ID":"db72d8fa-fc86-4654-b097-7d6a920a4d72","Type":"ContainerDied","Data":"62e19dc79eeb4ebde645808d442474418df1612b437bf3323155ff3c31d3cbbc"}
Oct 30 00:51:16.493479 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:16.493407    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-be862c05-6b54db7858-9bx9r" event={"ID":"db72d8fa-fc86-4654-b097-7d6a920a4d72","Type":"ContainerDied","Data":"323059f4cbc40ff280eec38c0aa8d2c2bb727b37986bc25e1b630c11d7ad2616"}
Oct 30 00:51:16.493479 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:16.493432    4393 scope.go:122] "RemoveContainer" containerID="62e19dc79eeb4ebde645808d442474418df1612b437bf3323155ff3c31d3cbbc"
Oct 30 00:51:16.495192 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:16.495168    4393 generic.go:356] "Generic (PLEG): container finished" podID="8e75c787-c583-464d-b087-be6f84901779" containerID="172afc7d67157c22d3235b5902e5c2f5b7864be601b4ede37a8e7e1c36acfbaa" exitCode=0
Oct 30 00:51:16.495301 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:16.495205    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-be862c05-c76879479-bfz2h"
Oct 30 00:51:16.495378 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:16.495320    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-be862c05-c76879479-bfz2h" event={"ID":"8e75c787-c583-464d-b087-be6f84901779","Type":"ContainerDied","Data":"172afc7d67157c22d3235b5902e5c2f5b7864be601b4ede37a8e7e1c36acfbaa"}
Oct 30 00:51:16.495378 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:16.495364    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-be862c05-c76879479-bfz2h" event={"ID":"8e75c787-c583-464d-b087-be6f84901779","Type":"ContainerDied","Data":"3ca5d88368eb972c7c7b481ab22790c18a7d221fff496afc67e37cdf2d20caba"}
Oct 30 00:51:16.507943 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:16.507879    4393 scope.go:122] "RemoveContainer" containerID="62e19dc79eeb4ebde645808d442474418df1612b437bf3323155ff3c31d3cbbc"
Oct 30 00:51:16.508233 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:51:16.508201    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62e19dc79eeb4ebde645808d442474418df1612b437bf3323155ff3c31d3cbbc\": container with ID starting with 62e19dc79eeb4ebde645808d442474418df1612b437bf3323155ff3c31d3cbbc not found: ID does not exist" containerID="62e19dc79eeb4ebde645808d442474418df1612b437bf3323155ff3c31d3cbbc"
Oct 30 00:51:16.508342 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:16.508243    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e19dc79eeb4ebde645808d442474418df1612b437bf3323155ff3c31d3cbbc"} err="failed to get container status \"62e19dc79eeb4ebde645808d442474418df1612b437bf3323155ff3c31d3cbbc\": rpc error: code = NotFound desc = could not find container \"62e19dc79eeb4ebde645808d442474418df1612b437bf3323155ff3c31d3cbbc\": container with ID starting with 62e19dc79eeb4ebde645808d442474418df1612b437bf3323155ff3c31d3cbbc not found: ID does not exist"
Oct 30 00:51:16.508342 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:16.508266    4393 scope.go:122] "RemoveContainer" containerID="172afc7d67157c22d3235b5902e5c2f5b7864be601b4ede37a8e7e1c36acfbaa"
Oct 30 00:51:16.522888 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:16.522794    4393 scope.go:122] "RemoveContainer" containerID="172afc7d67157c22d3235b5902e5c2f5b7864be601b4ede37a8e7e1c36acfbaa"
Oct 30 00:51:16.523245 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:51:16.523213    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"172afc7d67157c22d3235b5902e5c2f5b7864be601b4ede37a8e7e1c36acfbaa\": container with ID starting with 172afc7d67157c22d3235b5902e5c2f5b7864be601b4ede37a8e7e1c36acfbaa not found: ID does not exist" containerID="172afc7d67157c22d3235b5902e5c2f5b7864be601b4ede37a8e7e1c36acfbaa"
Oct 30 00:51:16.523330 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:16.523259    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"172afc7d67157c22d3235b5902e5c2f5b7864be601b4ede37a8e7e1c36acfbaa"} err="failed to get container status \"172afc7d67157c22d3235b5902e5c2f5b7864be601b4ede37a8e7e1c36acfbaa\": rpc error: code = NotFound desc = could not find container \"172afc7d67157c22d3235b5902e5c2f5b7864be601b4ede37a8e7e1c36acfbaa\": container with ID starting with 172afc7d67157c22d3235b5902e5c2f5b7864be601b4ede37a8e7e1c36acfbaa not found: ID does not exist"
Oct 30 00:51:16.547221 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:16.547138    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-be862c05-c76879479-bfz2h"]
Oct 30 00:51:16.552986 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:16.552957    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-be862c05-c76879479-bfz2h"]
Oct 30 00:51:16.568058 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:16.568030    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqrlc\" (UniqueName: \"kubernetes.io/projected/db72d8fa-fc86-4654-b097-7d6a920a4d72-kube-api-access-vqrlc\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:51:16.568058 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:16.568057    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcqnf\" (UniqueName: \"kubernetes.io/projected/8e75c787-c583-464d-b087-be6f84901779-kube-api-access-mcqnf\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:51:16.585516 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:16.585471    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-be862c05-6b54db7858-9bx9r"]
Oct 30 00:51:16.594367 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:16.594331    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-be862c05-6b54db7858-9bx9r"]
Oct 30 00:51:17.318858 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:17.318739    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5f387c-5cf86f7b99-9gvnl"]
Oct 30 00:51:17.319557 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:17.319540    4393 cpu_manager.go:414] "containerMap: removing container" podUID="8e75c787-c583-464d-b087-be6f84901779" containerName="payload-pre-processing"
Oct 30 00:51:17.319557 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:17.319557    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e75c787-c583-464d-b087-be6f84901779" containerName="payload-pre-processing"
Oct 30 00:51:17.319693 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:17.319608    4393 cpu_manager.go:414] "containerMap: removing container" podUID="db72d8fa-fc86-4654-b097-7d6a920a4d72" containerName="payload-processing"
Oct 30 00:51:17.319693 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:17.319614    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db72d8fa-fc86-4654-b097-7d6a920a4d72" containerName="payload-processing"
Oct 30 00:51:17.319693 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:17.319631    4393 cpu_manager.go:414] "containerMap: removing container" podUID="97b0fb2d-43b7-492d-b655-49c834f30bf8" containerName="maas-api"
Oct 30 00:51:17.319693 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:17.319638    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97b0fb2d-43b7-492d-b655-49c834f30bf8" containerName="maas-api"
Oct 30 00:51:17.319949 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:17.319745    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="db72d8fa-fc86-4654-b097-7d6a920a4d72" containerName="payload-processing"
Oct 30 00:51:17.319949 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:17.319760    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="97b0fb2d-43b7-492d-b655-49c834f30bf8" containerName="maas-api"
Oct 30 00:51:17.319949 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:17.319771    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e75c787-c583-464d-b087-be6f84901779" containerName="payload-pre-processing"
Oct 30 00:51:17.325091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:17.325071    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5f387c-5cf86f7b99-9gvnl"
Oct 30 00:51:17.344232 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:17.344208    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-5f387c\""
Oct 30 00:51:17.344365 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:17.344254    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-5f387c-dockercfg-gblrs\""
Oct 30 00:51:17.348109 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:17.348086    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5f387c-5cf86f7b99-9gvnl"]
Oct 30 00:51:17.379902 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:17.379871    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048e31e4-12d0-46ef-abae-1afebf63da75-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-5f387c-5cf86f7b99-9gvnl\" (UID: \"048e31e4-12d0-46ef-abae-1afebf63da75\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5f387c-5cf86f7b99-9gvnl"
Oct 30 00:51:17.380127 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:17.380109    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hm9l\" (UniqueName: \"kubernetes.io/projected/048e31e4-12d0-46ef-abae-1afebf63da75-kube-api-access-4hm9l\") pod \"payload-pre-processing-e2e-shared-a-w0-5f387c-5cf86f7b99-9gvnl\" (UID: \"048e31e4-12d0-46ef-abae-1afebf63da75\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5f387c-5cf86f7b99-9gvnl"
Oct 30 00:51:17.558373 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:17.556487    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5f387c-d78d85d96-f5csl"]
Oct 30 00:51:17.668209 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:17.668147    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hm9l\" (UniqueName: \"kubernetes.io/projected/048e31e4-12d0-46ef-abae-1afebf63da75-kube-api-access-4hm9l\") pod \"payload-pre-processing-e2e-shared-a-w0-5f387c-5cf86f7b99-9gvnl\" (UID: \"048e31e4-12d0-46ef-abae-1afebf63da75\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5f387c-5cf86f7b99-9gvnl"
Oct 30 00:51:17.668420 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:17.668331    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048e31e4-12d0-46ef-abae-1afebf63da75-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-5f387c-5cf86f7b99-9gvnl\" (UID: \"048e31e4-12d0-46ef-abae-1afebf63da75\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5f387c-5cf86f7b99-9gvnl"
Oct 30 00:51:17.669447 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:17.669414    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048e31e4-12d0-46ef-abae-1afebf63da75-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-5f387c-5cf86f7b99-9gvnl\" (UID: \"048e31e4-12d0-46ef-abae-1afebf63da75\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5f387c-5cf86f7b99-9gvnl"
Oct 30 00:51:17.673431 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:17.673395    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5f387c-d78d85d96-f5csl"
Oct 30 00:51:17.679972 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:17.679933    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5f387c-d78d85d96-f5csl"]
Oct 30 00:51:17.760851 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:17.759747    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hm9l\" (UniqueName: \"kubernetes.io/projected/048e31e4-12d0-46ef-abae-1afebf63da75-kube-api-access-4hm9l\") pod \"payload-pre-processing-e2e-shared-a-w0-5f387c-5cf86f7b99-9gvnl\" (UID: \"048e31e4-12d0-46ef-abae-1afebf63da75\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5f387c-5cf86f7b99-9gvnl"
Oct 30 00:51:17.785151 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:17.785117    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e75c787-c583-464d-b087-be6f84901779" path="/var/lib/kubelet/pods/8e75c787-c583-464d-b087-be6f84901779/volumes"
Oct 30 00:51:17.785856 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:17.785835    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db72d8fa-fc86-4654-b097-7d6a920a4d72" path="/var/lib/kubelet/pods/db72d8fa-fc86-4654-b097-7d6a920a4d72/volumes"
Oct 30 00:51:17.873348 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:17.873314    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bc74d4c-96b2-4227-9c2b-1145739bab41-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-5f387c-d78d85d96-f5csl\" (UID: \"1bc74d4c-96b2-4227-9c2b-1145739bab41\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5f387c-d78d85d96-f5csl"
Oct 30 00:51:17.873550 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:17.873364    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2bb7\" (UniqueName: \"kubernetes.io/projected/1bc74d4c-96b2-4227-9c2b-1145739bab41-kube-api-access-c2bb7\") pod \"payload-processing-e2e-shared-a-w0-5f387c-d78d85d96-f5csl\" (UID: \"1bc74d4c-96b2-4227-9c2b-1145739bab41\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5f387c-d78d85d96-f5csl"
Oct 30 00:51:17.939824 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:17.939714    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5f387c-5cf86f7b99-9gvnl"
Oct 30 00:51:17.974916 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:17.974889    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bc74d4c-96b2-4227-9c2b-1145739bab41-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-5f387c-d78d85d96-f5csl\" (UID: \"1bc74d4c-96b2-4227-9c2b-1145739bab41\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5f387c-d78d85d96-f5csl"
Oct 30 00:51:17.975073 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:17.974956    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2bb7\" (UniqueName: \"kubernetes.io/projected/1bc74d4c-96b2-4227-9c2b-1145739bab41-kube-api-access-c2bb7\") pod \"payload-processing-e2e-shared-a-w0-5f387c-d78d85d96-f5csl\" (UID: \"1bc74d4c-96b2-4227-9c2b-1145739bab41\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5f387c-d78d85d96-f5csl"
Oct 30 00:51:17.975621 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:17.975601    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bc74d4c-96b2-4227-9c2b-1145739bab41-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-5f387c-d78d85d96-f5csl\" (UID: \"1bc74d4c-96b2-4227-9c2b-1145739bab41\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5f387c-d78d85d96-f5csl"
Oct 30 00:51:17.999551 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:17.998399    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2bb7\" (UniqueName: \"kubernetes.io/projected/1bc74d4c-96b2-4227-9c2b-1145739bab41-kube-api-access-c2bb7\") pod \"payload-processing-e2e-shared-a-w0-5f387c-d78d85d96-f5csl\" (UID: \"1bc74d4c-96b2-4227-9c2b-1145739bab41\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5f387c-d78d85d96-f5csl"
Oct 30 00:51:18.078089 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:18.077982    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5f387c-d78d85d96-f5csl"
Oct 30 00:51:18.246621 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:51:18.246580    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod048e31e4_12d0_46ef_abae_1afebf63da75.slice/crio-7ab1f73a768871b9c309be479fdd8ffd86738df1c688e0f7fc2db631af55abfb WatchSource:0}: Error finding container 7ab1f73a768871b9c309be479fdd8ffd86738df1c688e0f7fc2db631af55abfb: Status 404 returned error can't find the container with id 7ab1f73a768871b9c309be479fdd8ffd86738df1c688e0f7fc2db631af55abfb
Oct 30 00:51:18.250158 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:18.249072    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5f387c-5cf86f7b99-9gvnl"]
Oct 30 00:51:18.403028 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:51:18.402997    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bc74d4c_96b2_4227_9c2b_1145739bab41.slice/crio-6a30224a470891a669f5ada881e6dee783a9404616eb5cd71779fb7246c54b0f WatchSource:0}: Error finding container 6a30224a470891a669f5ada881e6dee783a9404616eb5cd71779fb7246c54b0f: Status 404 returned error can't find the container with id 6a30224a470891a669f5ada881e6dee783a9404616eb5cd71779fb7246c54b0f
Oct 30 00:51:18.403733 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:18.403705    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5f387c-d78d85d96-f5csl"]
Oct 30 00:51:18.471394 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:18.471368    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52"
Oct 30 00:51:18.790520 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:18.790481    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5f387c-d78d85d96-f5csl" event={"ID":"1bc74d4c-96b2-4227-9c2b-1145739bab41","Type":"ContainerStarted","Data":"5e2c3f0acce6ee3db982c54d892c648e624080b6f0bd47ca3685fc729aaf9a3f"}
Oct 30 00:51:18.790706 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:18.790561    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5f387c-d78d85d96-f5csl" event={"ID":"1bc74d4c-96b2-4227-9c2b-1145739bab41","Type":"ContainerStarted","Data":"6a30224a470891a669f5ada881e6dee783a9404616eb5cd71779fb7246c54b0f"}
Oct 30 00:51:18.790706 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:18.790636    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5f387c-d78d85d96-f5csl"
Oct 30 00:51:18.792321 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:18.792286    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5f387c-5cf86f7b99-9gvnl" event={"ID":"048e31e4-12d0-46ef-abae-1afebf63da75","Type":"ContainerStarted","Data":"a10799c3c84c835c64b9a96de5aad694f99cf956443643a1a5c598da9c69e99d"}
Oct 30 00:51:18.792452 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:18.792330    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5f387c-5cf86f7b99-9gvnl" event={"ID":"048e31e4-12d0-46ef-abae-1afebf63da75","Type":"ContainerStarted","Data":"7ab1f73a768871b9c309be479fdd8ffd86738df1c688e0f7fc2db631af55abfb"}
Oct 30 00:51:18.792452 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:18.792432    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5f387c-5cf86f7b99-9gvnl"
Oct 30 00:51:18.832330 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:18.832280    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5f387c-d78d85d96-f5csl" podStartSLOduration=1.832266725 podStartE2EDuration="1.832266725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:18.830809351 +0000 UTC m=+1169.218485733" watchObservedRunningTime="2026-10-30 00:51:18.832266725 +0000 UTC m=+1169.219943114"
Oct 30 00:51:18.872514 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:18.872465    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5f387c-5cf86f7b99-9gvnl" podStartSLOduration=1.87244655 podStartE2EDuration="1.87244655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:18.868576587 +0000 UTC m=+1169.256252977" watchObservedRunningTime="2026-10-30 00:51:18.87244655 +0000 UTC m=+1169.260122941"
Oct 30 00:51:26.562701 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:51:26.562660    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:51:29.803436 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:29.803399    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5f387c-d78d85d96-f5csl"
Oct 30 00:51:29.803920 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:29.803470    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5f387c-5cf86f7b99-9gvnl"
Oct 30 00:51:31.943503 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:31.943472    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w"]
Oct 30 00:51:31.949531 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:31.949508    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w"
Oct 30 00:51:31.975082 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:31.974092    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w"]
Oct 30 00:51:31.978402 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:31.978366    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-5f387c-serving-cert\""
Oct 30 00:51:31.994297 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:31.994265    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf0d8c60-1725-48a4-b3e6-bfc144e13031-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w\" (UID: \"cf0d8c60-1725-48a4-b3e6-bfc144e13031\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w"
Oct 30 00:51:31.994468 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:31.994343    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf0d8c60-1725-48a4-b3e6-bfc144e13031-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w\" (UID: \"cf0d8c60-1725-48a4-b3e6-bfc144e13031\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w"
Oct 30 00:51:31.994543 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:31.994513    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjqld\" (UniqueName: \"kubernetes.io/projected/cf0d8c60-1725-48a4-b3e6-bfc144e13031-kube-api-access-xjqld\") pod \"maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w\" (UID: \"cf0d8c60-1725-48a4-b3e6-bfc144e13031\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w"
Oct 30 00:51:31.994762 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:31.994740    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf0d8c60-1725-48a4-b3e6-bfc144e13031-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w\" (UID: \"cf0d8c60-1725-48a4-b3e6-bfc144e13031\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w"
Oct 30 00:51:32.096128 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:32.096090    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf0d8c60-1725-48a4-b3e6-bfc144e13031-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w\" (UID: \"cf0d8c60-1725-48a4-b3e6-bfc144e13031\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w"
Oct 30 00:51:32.096324 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:32.096229    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf0d8c60-1725-48a4-b3e6-bfc144e13031-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w\" (UID: \"cf0d8c60-1725-48a4-b3e6-bfc144e13031\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w"
Oct 30 00:51:32.096324 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:32.096290    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf0d8c60-1725-48a4-b3e6-bfc144e13031-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w\" (UID: \"cf0d8c60-1725-48a4-b3e6-bfc144e13031\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w"
Oct 30 00:51:32.096455 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:32.096391    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjqld\" (UniqueName: \"kubernetes.io/projected/cf0d8c60-1725-48a4-b3e6-bfc144e13031-kube-api-access-xjqld\") pod \"maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w\" (UID: \"cf0d8c60-1725-48a4-b3e6-bfc144e13031\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w"
Oct 30 00:51:32.097327 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:32.097288    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf0d8c60-1725-48a4-b3e6-bfc144e13031-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w\" (UID: \"cf0d8c60-1725-48a4-b3e6-bfc144e13031\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w"
Oct 30 00:51:32.099037 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:32.099014    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf0d8c60-1725-48a4-b3e6-bfc144e13031-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w\" (UID: \"cf0d8c60-1725-48a4-b3e6-bfc144e13031\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w"
Oct 30 00:51:32.099037 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:32.099024    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf0d8c60-1725-48a4-b3e6-bfc144e13031-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w\" (UID: \"cf0d8c60-1725-48a4-b3e6-bfc144e13031\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w"
Oct 30 00:51:32.121135 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:32.121104    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjqld\" (UniqueName: \"kubernetes.io/projected/cf0d8c60-1725-48a4-b3e6-bfc144e13031-kube-api-access-xjqld\") pod \"maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w\" (UID: \"cf0d8c60-1725-48a4-b3e6-bfc144e13031\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w"
Oct 30 00:51:32.133858 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:32.133827    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-q78mf"]
Oct 30 00:51:32.139264 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:32.139244    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-q78mf"
Oct 30 00:51:32.153587 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:32.153560    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d7a6db25\""
Oct 30 00:51:32.153729 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:32.153706    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d7a6db25-dockercfg-7285h\""
Oct 30 00:51:32.157162 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:32.157139    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-q78mf"]
Oct 30 00:51:32.198343 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:32.198254    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br4zp\" (UniqueName: \"kubernetes.io/projected/999e3533-c8ce-4fff-a51d-1e7938955602-kube-api-access-br4zp\") pod \"payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-q78mf\" (UID: \"999e3533-c8ce-4fff-a51d-1e7938955602\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-q78mf"
Oct 30 00:51:32.198549 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:32.198527    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/999e3533-c8ce-4fff-a51d-1e7938955602-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-q78mf\" (UID: \"999e3533-c8ce-4fff-a51d-1e7938955602\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-q78mf"
Oct 30 00:51:32.282862 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:32.267097    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w"
Oct 30 00:51:32.351874 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:32.303243    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br4zp\" (UniqueName: \"kubernetes.io/projected/999e3533-c8ce-4fff-a51d-1e7938955602-kube-api-access-br4zp\") pod \"payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-q78mf\" (UID: \"999e3533-c8ce-4fff-a51d-1e7938955602\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-q78mf"
Oct 30 00:51:32.351874 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:32.303514    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/999e3533-c8ce-4fff-a51d-1e7938955602-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-q78mf\" (UID: \"999e3533-c8ce-4fff-a51d-1e7938955602\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-q78mf"
Oct 30 00:51:32.351874 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:32.304497    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/999e3533-c8ce-4fff-a51d-1e7938955602-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-q78mf\" (UID: \"999e3533-c8ce-4fff-a51d-1e7938955602\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-q78mf"
Oct 30 00:51:32.438558 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:32.438515    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br4zp\" (UniqueName: \"kubernetes.io/projected/999e3533-c8ce-4fff-a51d-1e7938955602-kube-api-access-br4zp\") pod \"payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-q78mf\" (UID: \"999e3533-c8ce-4fff-a51d-1e7938955602\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-q78mf"
Oct 30 00:51:32.473909 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:32.460801    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-q78mf"
Oct 30 00:51:32.851058 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:51:32.850992    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf0d8c60_1725_48a4_b3e6_bfc144e13031.slice/crio-06659f9d793792e6bc710ccb9d059f5dbda92121ceefa56f3df3f27f02b4ff63 WatchSource:0}: Error finding container 06659f9d793792e6bc710ccb9d059f5dbda92121ceefa56f3df3f27f02b4ff63: Status 404 returned error can't find the container with id 06659f9d793792e6bc710ccb9d059f5dbda92121ceefa56f3df3f27f02b4ff63
Oct 30 00:51:32.864471 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:32.864443    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w"]
Oct 30 00:51:32.903133 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:51:32.903107    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod999e3533_c8ce_4fff_a51d_1e7938955602.slice/crio-0957a88561743dc8eb0c4911522c23a78ff3163533ebfc5fd774fbd33d428065 WatchSource:0}: Error finding container 0957a88561743dc8eb0c4911522c23a78ff3163533ebfc5fd774fbd33d428065: Status 404 returned error can't find the container with id 0957a88561743dc8eb0c4911522c23a78ff3163533ebfc5fd774fbd33d428065
Oct 30 00:51:32.903817 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:32.903792    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-q78mf"]
Oct 30 00:51:33.482236 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:33.482200    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-q78mf" event={"ID":"999e3533-c8ce-4fff-a51d-1e7938955602","Type":"ContainerStarted","Data":"d4e0036339990c98261d3900f31a6ba033b706333d369cc902998548ff2531e0"}
Oct 30 00:51:33.482654 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:33.482268    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-q78mf" event={"ID":"999e3533-c8ce-4fff-a51d-1e7938955602","Type":"ContainerStarted","Data":"0957a88561743dc8eb0c4911522c23a78ff3163533ebfc5fd774fbd33d428065"}
Oct 30 00:51:33.482654 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:33.482526    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-q78mf"
Oct 30 00:51:33.485275 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:33.485240    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w" event={"ID":"cf0d8c60-1725-48a4-b3e6-bfc144e13031","Type":"ContainerStarted","Data":"06659f9d793792e6bc710ccb9d059f5dbda92121ceefa56f3df3f27f02b4ff63"}
Oct 30 00:51:33.522633 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:33.522570    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-q78mf" podStartSLOduration=1.522556472 podStartE2EDuration="1.522556472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:33.520589017 +0000 UTC m=+1183.908265406" watchObservedRunningTime="2026-10-30 00:51:33.522556472 +0000 UTC m=+1183.910232920"
Oct 30 00:51:34.495583 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:34.495545    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w" event={"ID":"cf0d8c60-1725-48a4-b3e6-bfc144e13031","Type":"ContainerStarted","Data":"b6d068ed946beb76dc52c9fcb834bd2ff685b46a04691da17ce5f58e220cd60a"}
Oct 30 00:51:34.496078 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:34.495800    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w"
Oct 30 00:51:34.539290 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:34.539239    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w" podStartSLOduration=3.08680968 podStartE2EDuration="3.53922542s" totalImagesPullingTime="452.41574ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:34.538133496 +0000 UTC m=+1184.925809886" watchObservedRunningTime="2026-10-30 00:51:34.53922542 +0000 UTC m=+1184.926901870"
Oct 30 00:51:40.527124 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:40.527097    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w"
Oct 30 00:51:41.001772 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:41.001733    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e0f66-fc8567cdc-knftm"]
Oct 30 00:51:41.009543 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:41.008558    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e0f66-fc8567cdc-knftm"
Oct 30 00:51:41.052431 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:41.052400    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-8e0f66\""
Oct 30 00:51:41.226848 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:41.199556    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bfbd1a2-2a38-43ed-abe0-c1a719454e10-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-8e0f66-fc8567cdc-knftm\" (UID: \"6bfbd1a2-2a38-43ed-abe0-c1a719454e10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e0f66-fc8567cdc-knftm"
Oct 30 00:51:41.226848 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:41.199770    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cxc5\" (UniqueName: \"kubernetes.io/projected/6bfbd1a2-2a38-43ed-abe0-c1a719454e10-kube-api-access-4cxc5\") pod \"payload-pre-processing-e2e-shared-b-w3-8e0f66-fc8567cdc-knftm\" (UID: \"6bfbd1a2-2a38-43ed-abe0-c1a719454e10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e0f66-fc8567cdc-knftm"
Oct 30 00:51:41.310602 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:41.310542    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cxc5\" (UniqueName: \"kubernetes.io/projected/6bfbd1a2-2a38-43ed-abe0-c1a719454e10-kube-api-access-4cxc5\") pod \"payload-pre-processing-e2e-shared-b-w3-8e0f66-fc8567cdc-knftm\" (UID: \"6bfbd1a2-2a38-43ed-abe0-c1a719454e10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e0f66-fc8567cdc-knftm"
Oct 30 00:51:41.325178 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:41.325134    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bfbd1a2-2a38-43ed-abe0-c1a719454e10-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-8e0f66-fc8567cdc-knftm\" (UID: \"6bfbd1a2-2a38-43ed-abe0-c1a719454e10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e0f66-fc8567cdc-knftm"
Oct 30 00:51:41.357611 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:41.356449    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bfbd1a2-2a38-43ed-abe0-c1a719454e10-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-8e0f66-fc8567cdc-knftm\" (UID: \"6bfbd1a2-2a38-43ed-abe0-c1a719454e10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e0f66-fc8567cdc-knftm"
Oct 30 00:51:41.360310 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:41.360280    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e0f66-fc8567cdc-knftm"]
Oct 30 00:51:41.362829 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:41.362765    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cxc5\" (UniqueName: \"kubernetes.io/projected/6bfbd1a2-2a38-43ed-abe0-c1a719454e10-kube-api-access-4cxc5\") pod \"payload-pre-processing-e2e-shared-b-w3-8e0f66-fc8567cdc-knftm\" (UID: \"6bfbd1a2-2a38-43ed-abe0-c1a719454e10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e0f66-fc8567cdc-knftm"
Oct 30 00:51:41.365450 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:41.365425    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e0f66-fc8567cdc-knftm"
Oct 30 00:51:41.367000 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:41.366480    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8e0f66-8646cb7ccd-fw5mb"]
Oct 30 00:51:41.376029 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:41.376007    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e0f66-8646cb7ccd-fw5mb"
Oct 30 00:51:41.378025 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:41.377999    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8e0f66-8646cb7ccd-fw5mb"]
Oct 30 00:51:41.390387 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:41.390360    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-8e0f66-dockercfg-98bbx\""
Oct 30 00:51:41.457547 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:41.457518    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3847bedb-934b-4dac-a5c1-7e5d593ba2ec-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-8e0f66-8646cb7ccd-fw5mb\" (UID: \"3847bedb-934b-4dac-a5c1-7e5d593ba2ec\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e0f66-8646cb7ccd-fw5mb"
Oct 30 00:51:41.457697 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:41.457569    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67jrz\" (UniqueName: \"kubernetes.io/projected/3847bedb-934b-4dac-a5c1-7e5d593ba2ec-kube-api-access-67jrz\") pod \"payload-processing-e2e-shared-b-w3-8e0f66-8646cb7ccd-fw5mb\" (UID: \"3847bedb-934b-4dac-a5c1-7e5d593ba2ec\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e0f66-8646cb7ccd-fw5mb"
Oct 30 00:51:41.559829 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:41.559734    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3847bedb-934b-4dac-a5c1-7e5d593ba2ec-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-8e0f66-8646cb7ccd-fw5mb\" (UID: \"3847bedb-934b-4dac-a5c1-7e5d593ba2ec\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e0f66-8646cb7ccd-fw5mb"
Oct 30 00:51:41.559829 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:41.559797    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67jrz\" (UniqueName: \"kubernetes.io/projected/3847bedb-934b-4dac-a5c1-7e5d593ba2ec-kube-api-access-67jrz\") pod \"payload-processing-e2e-shared-b-w3-8e0f66-8646cb7ccd-fw5mb\" (UID: \"3847bedb-934b-4dac-a5c1-7e5d593ba2ec\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e0f66-8646cb7ccd-fw5mb"
Oct 30 00:51:41.560518 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:41.560499    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3847bedb-934b-4dac-a5c1-7e5d593ba2ec-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-8e0f66-8646cb7ccd-fw5mb\" (UID: \"3847bedb-934b-4dac-a5c1-7e5d593ba2ec\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e0f66-8646cb7ccd-fw5mb"
Oct 30 00:51:41.582923 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:41.582893    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67jrz\" (UniqueName: \"kubernetes.io/projected/3847bedb-934b-4dac-a5c1-7e5d593ba2ec-kube-api-access-67jrz\") pod \"payload-processing-e2e-shared-b-w3-8e0f66-8646cb7ccd-fw5mb\" (UID: \"3847bedb-934b-4dac-a5c1-7e5d593ba2ec\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e0f66-8646cb7ccd-fw5mb"
Oct 30 00:51:41.678134 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:51:41.678103    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bfbd1a2_2a38_43ed_abe0_c1a719454e10.slice/crio-769df99ac61d10fb86be021f80b171d3a7fb1e5aea5ef93ac046a20c38c80acb WatchSource:0}: Error finding container 769df99ac61d10fb86be021f80b171d3a7fb1e5aea5ef93ac046a20c38c80acb: Status 404 returned error can't find the container with id 769df99ac61d10fb86be021f80b171d3a7fb1e5aea5ef93ac046a20c38c80acb
Oct 30 00:51:41.678848 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:41.678818    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e0f66-fc8567cdc-knftm"]
Oct 30 00:51:41.722574 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:41.722551    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e0f66-8646cb7ccd-fw5mb"
Oct 30 00:51:42.043585 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:42.043552    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8e0f66-8646cb7ccd-fw5mb"]
Oct 30 00:51:42.595422 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:42.595382    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e0f66-8646cb7ccd-fw5mb" event={"ID":"3847bedb-934b-4dac-a5c1-7e5d593ba2ec","Type":"ContainerStarted","Data":"1b3c77ad94471d5ade787d6731ff467e28ad8977ba5d80e6b7d6a8f7aa352d5b"}
Oct 30 00:51:42.595897 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:42.595438    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e0f66-8646cb7ccd-fw5mb" event={"ID":"3847bedb-934b-4dac-a5c1-7e5d593ba2ec","Type":"ContainerStarted","Data":"e87b18ce6c1a24ba6268cafb05ca601ee9723763381d56e3ebad91b8b00460ae"}
Oct 30 00:51:42.595897 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:42.595464    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e0f66-8646cb7ccd-fw5mb"
Oct 30 00:51:42.597087 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:42.597063    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e0f66-fc8567cdc-knftm" event={"ID":"6bfbd1a2-2a38-43ed-abe0-c1a719454e10","Type":"ContainerStarted","Data":"a83eaebac0283dda84c234a6686f1efd9554b97a22c2b769d1b9e3136f65766a"}
Oct 30 00:51:42.597197 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:42.597103    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e0f66-fc8567cdc-knftm" event={"ID":"6bfbd1a2-2a38-43ed-abe0-c1a719454e10","Type":"ContainerStarted","Data":"769df99ac61d10fb86be021f80b171d3a7fb1e5aea5ef93ac046a20c38c80acb"}
Oct 30 00:51:42.597197 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:42.597148    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e0f66-fc8567cdc-knftm"
Oct 30 00:51:42.636130 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:42.636082    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e0f66-8646cb7ccd-fw5mb" podStartSLOduration=1.636068514 podStartE2EDuration="1.636068514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:42.633829709 +0000 UTC m=+1193.021506095" watchObservedRunningTime="2026-10-30 00:51:42.636068514 +0000 UTC m=+1193.023744923"
Oct 30 00:51:42.675014 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:42.674961    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e0f66-fc8567cdc-knftm" podStartSLOduration=2.674947006 podStartE2EDuration="2.674947006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:42.672135624 +0000 UTC m=+1193.059812013" watchObservedRunningTime="2026-10-30 00:51:42.674947006 +0000 UTC m=+1193.062623395"
Oct 30 00:51:44.500034 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:44.500005    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-q78mf"
Oct 30 00:51:45.826338 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:45.826306    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786"]
Oct 30 00:51:45.830796 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:45.830758    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786"
Oct 30 00:51:45.882277 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:45.879865    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786"]
Oct 30 00:51:45.885678 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:45.885649    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d7a6db25-serving-cert\""
Oct 30 00:51:45.927341 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:45.927309    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgtfn\" (UniqueName: \"kubernetes.io/projected/6f2d0713-1871-487b-857e-f10b16ad2622-kube-api-access-xgtfn\") pod \"maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786\" (UID: \"6f2d0713-1871-487b-857e-f10b16ad2622\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786"
Oct 30 00:51:45.927544 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:45.927521    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2d0713-1871-487b-857e-f10b16ad2622-openshift-service-ca\") pod \"maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786\" (UID: \"6f2d0713-1871-487b-857e-f10b16ad2622\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786"
Oct 30 00:51:45.927632 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:45.927563    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f2d0713-1871-487b-857e-f10b16ad2622-maas-api-tls\") pod \"maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786\" (UID: \"6f2d0713-1871-487b-857e-f10b16ad2622\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786"
Oct 30 00:51:45.927850 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:45.927834    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f2d0713-1871-487b-857e-f10b16ad2622-metrics-certs\") pod \"maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786\" (UID: \"6f2d0713-1871-487b-857e-f10b16ad2622\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786"
Oct 30 00:51:46.029648 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:46.029611    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f2d0713-1871-487b-857e-f10b16ad2622-metrics-certs\") pod \"maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786\" (UID: \"6f2d0713-1871-487b-857e-f10b16ad2622\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786"
Oct 30 00:51:46.029840 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:46.029678    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgtfn\" (UniqueName: \"kubernetes.io/projected/6f2d0713-1871-487b-857e-f10b16ad2622-kube-api-access-xgtfn\") pod \"maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786\" (UID: \"6f2d0713-1871-487b-857e-f10b16ad2622\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786"
Oct 30 00:51:46.029840 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:46.029799    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2d0713-1871-487b-857e-f10b16ad2622-openshift-service-ca\") pod \"maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786\" (UID: \"6f2d0713-1871-487b-857e-f10b16ad2622\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786"
Oct 30 00:51:46.029840 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:46.029827    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f2d0713-1871-487b-857e-f10b16ad2622-maas-api-tls\") pod \"maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786\" (UID: \"6f2d0713-1871-487b-857e-f10b16ad2622\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786"
Oct 30 00:51:46.030582 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:46.030553    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2d0713-1871-487b-857e-f10b16ad2622-openshift-service-ca\") pod \"maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786\" (UID: \"6f2d0713-1871-487b-857e-f10b16ad2622\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786"
Oct 30 00:51:46.032429 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:46.032399    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f2d0713-1871-487b-857e-f10b16ad2622-maas-api-tls\") pod \"maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786\" (UID: \"6f2d0713-1871-487b-857e-f10b16ad2622\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786"
Oct 30 00:51:46.032545 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:46.032475    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f2d0713-1871-487b-857e-f10b16ad2622-metrics-certs\") pod \"maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786\" (UID: \"6f2d0713-1871-487b-857e-f10b16ad2622\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786"
Oct 30 00:51:46.104849 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:46.102450    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgtfn\" (UniqueName: \"kubernetes.io/projected/6f2d0713-1871-487b-857e-f10b16ad2622-kube-api-access-xgtfn\") pod \"maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786\" (UID: \"6f2d0713-1871-487b-857e-f10b16ad2622\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786"
Oct 30 00:51:46.146636 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:46.146484    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786"
Oct 30 00:51:46.516091 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:51:46.516059    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f2d0713_1871_487b_857e_f10b16ad2622.slice/crio-a040b7969a8fc3298d2c3aba549ddd07b6bf892b0ee9f2ccf1c185c44d41bbb6 WatchSource:0}: Error finding container a040b7969a8fc3298d2c3aba549ddd07b6bf892b0ee9f2ccf1c185c44d41bbb6: Status 404 returned error can't find the container with id a040b7969a8fc3298d2c3aba549ddd07b6bf892b0ee9f2ccf1c185c44d41bbb6
Oct 30 00:51:46.516664 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:46.516641    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786"]
Oct 30 00:51:46.696863 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:46.696819    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786" event={"ID":"6f2d0713-1871-487b-857e-f10b16ad2622","Type":"ContainerStarted","Data":"a040b7969a8fc3298d2c3aba549ddd07b6bf892b0ee9f2ccf1c185c44d41bbb6"}
Oct 30 00:51:47.803906 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:47.803868    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786" event={"ID":"6f2d0713-1871-487b-857e-f10b16ad2622","Type":"ContainerStarted","Data":"db71b10d07acf584e8cf986e47c254affd9e7fc467a13464787c01f80192faf8"}
Oct 30 00:51:47.804733 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:47.804716    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786"
Oct 30 00:51:47.846067 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:47.846007    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786" podStartSLOduration=2.537898782 podStartE2EDuration="2.845987349s" totalImagesPullingTime="308.088567ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:51:47.844372703 +0000 UTC m=+1198.232049094" watchObservedRunningTime="2026-10-30 00:51:47.845987349 +0000 UTC m=+1198.233663739"
Oct 30 00:51:49.302919 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:49.302875    4393 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-56.ec2.internal" podUID="21539217511a9b702eb77d59ab0b339d" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:53.064955 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:53.064901    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q" podUID="46b01388-7da7-4e63-95f3-afc811bed013" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:53.089660 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:53.089612    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g" podUID="fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:53.114091 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:53.112554    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x" podUID="28f3e4ff-5875-4e5d-8cab-60d1a5ba97be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:53.612855 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:53.610039    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e0f66-8646cb7ccd-fw5mb"
Oct 30 00:51:53.615601 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:53.615573    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e0f66-fc8567cdc-knftm"
Oct 30 00:51:54.884177 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:54.884142    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786"
Oct 30 00:51:55.565926 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:55.565884    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kfpcv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:51:55.566247 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:55.566215    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv" podUID="a5b4591e-8713-40c5-b33c-e6505356ee29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:51:56.513332 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:51:56.513288    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:51:59.407205 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:59.407172    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx"]
Oct 30 00:51:59.431263 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:59.430857    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx"
Oct 30 00:51:59.440535 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:59.440500    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx"]
Oct 30 00:51:59.452977 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:59.452946    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-8e0f66-serving-cert\""
Oct 30 00:51:59.507454 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:59.507423    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7416ca5a-6ff8-4937-84cf-15a156c523eb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx\" (UID: \"7416ca5a-6ff8-4937-84cf-15a156c523eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx"
Oct 30 00:51:59.507687 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:59.507668    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktfrn\" (UniqueName: \"kubernetes.io/projected/7416ca5a-6ff8-4937-84cf-15a156c523eb-kube-api-access-ktfrn\") pod \"maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx\" (UID: \"7416ca5a-6ff8-4937-84cf-15a156c523eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx"
Oct 30 00:51:59.507728 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:59.507702    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7416ca5a-6ff8-4937-84cf-15a156c523eb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx\" (UID: \"7416ca5a-6ff8-4937-84cf-15a156c523eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx"
Oct 30 00:51:59.507837 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:59.507811    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7416ca5a-6ff8-4937-84cf-15a156c523eb-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx\" (UID: \"7416ca5a-6ff8-4937-84cf-15a156c523eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx"
Oct 30 00:51:59.609157 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:59.609120    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7416ca5a-6ff8-4937-84cf-15a156c523eb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx\" (UID: \"7416ca5a-6ff8-4937-84cf-15a156c523eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx"
Oct 30 00:51:59.609326 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:59.609285    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktfrn\" (UniqueName: \"kubernetes.io/projected/7416ca5a-6ff8-4937-84cf-15a156c523eb-kube-api-access-ktfrn\") pod \"maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx\" (UID: \"7416ca5a-6ff8-4937-84cf-15a156c523eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx"
Oct 30 00:51:59.609326 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:59.609309    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7416ca5a-6ff8-4937-84cf-15a156c523eb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx\" (UID: \"7416ca5a-6ff8-4937-84cf-15a156c523eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx"
Oct 30 00:51:59.609414 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:59.609339    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7416ca5a-6ff8-4937-84cf-15a156c523eb-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx\" (UID: \"7416ca5a-6ff8-4937-84cf-15a156c523eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx"
Oct 30 00:51:59.610041 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:59.610009    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7416ca5a-6ff8-4937-84cf-15a156c523eb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx\" (UID: \"7416ca5a-6ff8-4937-84cf-15a156c523eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx"
Oct 30 00:51:59.611760 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:59.611739    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7416ca5a-6ff8-4937-84cf-15a156c523eb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx\" (UID: \"7416ca5a-6ff8-4937-84cf-15a156c523eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx"
Oct 30 00:51:59.611857 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:59.611793    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7416ca5a-6ff8-4937-84cf-15a156c523eb-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx\" (UID: \"7416ca5a-6ff8-4937-84cf-15a156c523eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx"
Oct 30 00:51:59.658882 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:59.658772    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktfrn\" (UniqueName: \"kubernetes.io/projected/7416ca5a-6ff8-4937-84cf-15a156c523eb-kube-api-access-ktfrn\") pod \"maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx\" (UID: \"7416ca5a-6ff8-4937-84cf-15a156c523eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx"
Oct 30 00:51:59.759951 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:51:59.759919    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx"
Oct 30 00:52:00.110690 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:52:00.110658    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7416ca5a_6ff8_4937_84cf_15a156c523eb.slice/crio-3ae242af21d20f74af14a1d7d3bc3a9e62e73bd6143d348b0d6f09103ae26479 WatchSource:0}: Error finding container 3ae242af21d20f74af14a1d7d3bc3a9e62e73bd6143d348b0d6f09103ae26479: Status 404 returned error can't find the container with id 3ae242af21d20f74af14a1d7d3bc3a9e62e73bd6143d348b0d6f09103ae26479
Oct 30 00:52:00.110865 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:00.110844    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx"]
Oct 30 00:52:01.079351 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:01.079309    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx" event={"ID":"7416ca5a-6ff8-4937-84cf-15a156c523eb","Type":"ContainerStarted","Data":"b92dee2a8d518ec864eaa875cf36a3210af817ca720fef3515a0745790c3e17a"}
Oct 30 00:52:01.079861 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:01.079381    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx" event={"ID":"7416ca5a-6ff8-4937-84cf-15a156c523eb","Type":"ContainerStarted","Data":"3ae242af21d20f74af14a1d7d3bc3a9e62e73bd6143d348b0d6f09103ae26479"}
Oct 30 00:52:01.079861 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:01.079580    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx"
Oct 30 00:52:01.141304 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:01.140721    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx" podStartSLOduration=1.796426529 podStartE2EDuration="2.140695077s" totalImagesPullingTime="344.268548ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:51:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:01.127088535 +0000 UTC m=+1211.514765045" watchObservedRunningTime="2026-10-30 00:52:01.140695077 +0000 UTC m=+1211.528371472"
Oct 30 00:52:07.093491 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:07.093459    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx"
Oct 30 00:52:22.167297 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:22.167264    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk"]
Oct 30 00:52:22.172312 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:22.172294    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk"
Oct 30 00:52:22.193322 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:22.193291    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk"]
Oct 30 00:52:22.197042 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:22.197011    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-00e07a-serving-cert\""
Oct 30 00:52:22.256466 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:22.256415    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxw5p\" (UniqueName: \"kubernetes.io/projected/f72f3812-276c-4a44-af20-917bc1e32a3f-kube-api-access-kxw5p\") pod \"maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk\" (UID: \"f72f3812-276c-4a44-af20-917bc1e32a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk"
Oct 30 00:52:22.256641 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:22.256588    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f72f3812-276c-4a44-af20-917bc1e32a3f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk\" (UID: \"f72f3812-276c-4a44-af20-917bc1e32a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk"
Oct 30 00:52:22.256641 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:22.256626    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f72f3812-276c-4a44-af20-917bc1e32a3f-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk\" (UID: \"f72f3812-276c-4a44-af20-917bc1e32a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk"
Oct 30 00:52:22.256852 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:22.256830    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f72f3812-276c-4a44-af20-917bc1e32a3f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk\" (UID: \"f72f3812-276c-4a44-af20-917bc1e32a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk"
Oct 30 00:52:22.362450 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:22.359174    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f72f3812-276c-4a44-af20-917bc1e32a3f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk\" (UID: \"f72f3812-276c-4a44-af20-917bc1e32a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk"
Oct 30 00:52:22.362450 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:22.359230    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f72f3812-276c-4a44-af20-917bc1e32a3f-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk\" (UID: \"f72f3812-276c-4a44-af20-917bc1e32a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk"
Oct 30 00:52:22.362450 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:22.359334    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f72f3812-276c-4a44-af20-917bc1e32a3f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk\" (UID: \"f72f3812-276c-4a44-af20-917bc1e32a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk"
Oct 30 00:52:22.362450 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:22.359695    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxw5p\" (UniqueName: \"kubernetes.io/projected/f72f3812-276c-4a44-af20-917bc1e32a3f-kube-api-access-kxw5p\") pod \"maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk\" (UID: \"f72f3812-276c-4a44-af20-917bc1e32a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk"
Oct 30 00:52:22.362450 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:52:22.360383    4393 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-00e07a-serving-cert: secret "maas-api-e2e-shared-b-w0-00e07a-serving-cert" not found
Oct 30 00:52:22.362450 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:52:22.360448    4393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f72f3812-276c-4a44-af20-917bc1e32a3f-maas-api-tls podName:f72f3812-276c-4a44-af20-917bc1e32a3f nodeName:}" failed. No retries permitted until 2026-10-30 00:52:22.860426908 +0000 UTC m=+1233.248103274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f72f3812-276c-4a44-af20-917bc1e32a3f-maas-api-tls") pod "maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk" (UID: "f72f3812-276c-4a44-af20-917bc1e32a3f") : secret "maas-api-e2e-shared-b-w0-00e07a-serving-cert" not found
Oct 30 00:52:22.397808 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:22.397466    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f72f3812-276c-4a44-af20-917bc1e32a3f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk\" (UID: \"f72f3812-276c-4a44-af20-917bc1e32a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk"
Oct 30 00:52:22.428849 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:22.425632    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxw5p\" (UniqueName: \"kubernetes.io/projected/f72f3812-276c-4a44-af20-917bc1e32a3f-kube-api-access-kxw5p\") pod \"maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk\" (UID: \"f72f3812-276c-4a44-af20-917bc1e32a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk"
Oct 30 00:52:22.436802 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:22.429740    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f72f3812-276c-4a44-af20-917bc1e32a3f-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk\" (UID: \"f72f3812-276c-4a44-af20-917bc1e32a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk"
Oct 30 00:52:22.871573 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:22.871546    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f72f3812-276c-4a44-af20-917bc1e32a3f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk\" (UID: \"f72f3812-276c-4a44-af20-917bc1e32a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk"
Oct 30 00:52:22.874178 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:22.874154    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f72f3812-276c-4a44-af20-917bc1e32a3f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk\" (UID: \"f72f3812-276c-4a44-af20-917bc1e32a3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk"
Oct 30 00:52:23.092202 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:23.092157    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk"
Oct 30 00:52:23.412661 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:23.412567    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk"]
Oct 30 00:52:23.800959 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:23.800884    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"]
Oct 30 00:52:23.807361 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:23.807275    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"
Oct 30 00:52:23.837992 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:23.837961    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"]
Oct 30 00:52:23.896057 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:23.896029    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9905328c-20ac-42d2-a90e-e2190360b082-istio-podinfo\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv\" (UID: \"9905328c-20ac-42d2-a90e-e2190360b082\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"
Oct 30 00:52:23.896169 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:23.896072    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x459r\" (UniqueName: \"kubernetes.io/projected/9905328c-20ac-42d2-a90e-e2190360b082-kube-api-access-x459r\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv\" (UID: \"9905328c-20ac-42d2-a90e-e2190360b082\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"
Oct 30 00:52:23.896169 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:23.896094    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9905328c-20ac-42d2-a90e-e2190360b082-istio-token\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv\" (UID: \"9905328c-20ac-42d2-a90e-e2190360b082\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"
Oct 30 00:52:23.896416 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:23.896382    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9905328c-20ac-42d2-a90e-e2190360b082-istio-data\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv\" (UID: \"9905328c-20ac-42d2-a90e-e2190360b082\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"
Oct 30 00:52:23.896526 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:23.896482    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9905328c-20ac-42d2-a90e-e2190360b082-istiod-ca-cert\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv\" (UID: \"9905328c-20ac-42d2-a90e-e2190360b082\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"
Oct 30 00:52:23.896583 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:23.896565    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9905328c-20ac-42d2-a90e-e2190360b082-workload-certs\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv\" (UID: \"9905328c-20ac-42d2-a90e-e2190360b082\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"
Oct 30 00:52:23.896629 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:23.896595    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9905328c-20ac-42d2-a90e-e2190360b082-istio-envoy\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv\" (UID: \"9905328c-20ac-42d2-a90e-e2190360b082\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"
Oct 30 00:52:23.896811 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:23.896793    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9905328c-20ac-42d2-a90e-e2190360b082-credential-socket\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv\" (UID: \"9905328c-20ac-42d2-a90e-e2190360b082\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"
Oct 30 00:52:23.896879 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:23.896836    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9905328c-20ac-42d2-a90e-e2190360b082-workload-socket\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv\" (UID: \"9905328c-20ac-42d2-a90e-e2190360b082\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"
Oct 30 00:52:23.909902 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:23.909882    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d7a6db25-openshift-default-dockercfg-qnm52\""
Oct 30 00:52:23.998231 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:23.998190    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9905328c-20ac-42d2-a90e-e2190360b082-credential-socket\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv\" (UID: \"9905328c-20ac-42d2-a90e-e2190360b082\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"
Oct 30 00:52:23.998364 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:23.998243    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9905328c-20ac-42d2-a90e-e2190360b082-workload-socket\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv\" (UID: \"9905328c-20ac-42d2-a90e-e2190360b082\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"
Oct 30 00:52:23.998364 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:23.998295    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9905328c-20ac-42d2-a90e-e2190360b082-istio-podinfo\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv\" (UID: \"9905328c-20ac-42d2-a90e-e2190360b082\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"
Oct 30 00:52:23.998364 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:23.998330    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x459r\" (UniqueName: \"kubernetes.io/projected/9905328c-20ac-42d2-a90e-e2190360b082-kube-api-access-x459r\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv\" (UID: \"9905328c-20ac-42d2-a90e-e2190360b082\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"
Oct 30 00:52:23.998539 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:23.998366    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9905328c-20ac-42d2-a90e-e2190360b082-istio-token\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv\" (UID: \"9905328c-20ac-42d2-a90e-e2190360b082\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"
Oct 30 00:52:23.998599 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:23.998567    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9905328c-20ac-42d2-a90e-e2190360b082-istio-data\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv\" (UID: \"9905328c-20ac-42d2-a90e-e2190360b082\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"
Oct 30 00:52:23.998681 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:23.998605    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9905328c-20ac-42d2-a90e-e2190360b082-istiod-ca-cert\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv\" (UID: \"9905328c-20ac-42d2-a90e-e2190360b082\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"
Oct 30 00:52:23.998681 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:23.998626    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9905328c-20ac-42d2-a90e-e2190360b082-credential-socket\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv\" (UID: \"9905328c-20ac-42d2-a90e-e2190360b082\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"
Oct 30 00:52:23.998681 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:23.998638    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9905328c-20ac-42d2-a90e-e2190360b082-workload-certs\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv\" (UID: \"9905328c-20ac-42d2-a90e-e2190360b082\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"
Oct 30 00:52:23.998681 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:23.998627    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9905328c-20ac-42d2-a90e-e2190360b082-workload-socket\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv\" (UID: \"9905328c-20ac-42d2-a90e-e2190360b082\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"
Oct 30 00:52:23.998681 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:23.998663    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9905328c-20ac-42d2-a90e-e2190360b082-istio-envoy\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv\" (UID: \"9905328c-20ac-42d2-a90e-e2190360b082\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"
Oct 30 00:52:23.998992 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:23.998880    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9905328c-20ac-42d2-a90e-e2190360b082-istio-data\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv\" (UID: \"9905328c-20ac-42d2-a90e-e2190360b082\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"
Oct 30 00:52:23.999121 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:23.999095    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9905328c-20ac-42d2-a90e-e2190360b082-workload-certs\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv\" (UID: \"9905328c-20ac-42d2-a90e-e2190360b082\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"
Oct 30 00:52:23.999231 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:23.999208    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9905328c-20ac-42d2-a90e-e2190360b082-istiod-ca-cert\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv\" (UID: \"9905328c-20ac-42d2-a90e-e2190360b082\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"
Oct 30 00:52:24.001346 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:24.001317    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9905328c-20ac-42d2-a90e-e2190360b082-istio-podinfo\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv\" (UID: \"9905328c-20ac-42d2-a90e-e2190360b082\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"
Oct 30 00:52:24.001462 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:24.001381    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9905328c-20ac-42d2-a90e-e2190360b082-istio-envoy\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv\" (UID: \"9905328c-20ac-42d2-a90e-e2190360b082\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"
Oct 30 00:52:24.021163 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:24.021137    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x459r\" (UniqueName: \"kubernetes.io/projected/9905328c-20ac-42d2-a90e-e2190360b082-kube-api-access-x459r\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv\" (UID: \"9905328c-20ac-42d2-a90e-e2190360b082\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"
Oct 30 00:52:24.021300 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:24.021144    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9905328c-20ac-42d2-a90e-e2190360b082-istio-token\") pod \"e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv\" (UID: \"9905328c-20ac-42d2-a90e-e2190360b082\") " pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"
Oct 30 00:52:24.134566 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:24.134537    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"
Oct 30 00:52:24.460906 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:52:24.460862    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9905328c_20ac_42d2_a90e_e2190360b082.slice/crio-cce95a327873a589dc5e861bd8c5f03b5133ddcbe43dcc760d6b3d73e695cc87 WatchSource:0}: Error finding container cce95a327873a589dc5e861bd8c5f03b5133ddcbe43dcc760d6b3d73e695cc87: Status 404 returned error can't find the container with id cce95a327873a589dc5e861bd8c5f03b5133ddcbe43dcc760d6b3d73e695cc87
Oct 30 00:52:24.461511 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:24.461488    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"]
Oct 30 00:52:24.464419 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:24.464380    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:52:24.464537 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:24.464453    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:52:24.464537 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:24.464483    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:52:24.565883 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:24.565832    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv" event={"ID":"9905328c-20ac-42d2-a90e-e2190360b082","Type":"ContainerStarted","Data":"a5c434c78e2dfe5d609f4687f3f98626aad6ee32cac4edace135ddb6518b4c09"}
Oct 30 00:52:24.566057 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:24.565901    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv" event={"ID":"9905328c-20ac-42d2-a90e-e2190360b082","Type":"ContainerStarted","Data":"cce95a327873a589dc5e861bd8c5f03b5133ddcbe43dcc760d6b3d73e695cc87"}
Oct 30 00:52:24.568047 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:24.568011    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk" event={"ID":"f72f3812-276c-4a44-af20-917bc1e32a3f","Type":"ContainerStarted","Data":"ceefd0a93fea8d5683bdc6344f55688cd3fc800a1db9815895b3fd3b6c623391"}
Oct 30 00:52:24.568177 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:24.568075    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk" event={"ID":"f72f3812-276c-4a44-af20-917bc1e32a3f","Type":"ContainerStarted","Data":"9eb49ef2827e7568b937619ecd6ffea16b36ea6a5695c8942aa56f6b79bad70c"}
Oct 30 00:52:24.568177 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:24.568120    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk"
Oct 30 00:52:24.616150 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:24.616095    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv" podStartSLOduration=1.616075242 podStartE2EDuration="1.616075242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:24.609909116 +0000 UTC m=+1234.997585520" watchObservedRunningTime="2026-10-30 00:52:24.616075242 +0000 UTC m=+1235.003751633"
Oct 30 00:52:24.652678 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:24.652620    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk" podStartSLOduration=2.287403309 podStartE2EDuration="2.652606921s" totalImagesPullingTime="365.203612ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:24.650505844 +0000 UTC m=+1235.038182234" watchObservedRunningTime="2026-10-30 00:52:24.652606921 +0000 UTC m=+1235.040283311"
Oct 30 00:52:25.135233 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:25.135204    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"
Oct 30 00:52:25.137107 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:25.137081    4393 patch_prober.go:29] interesting pod/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 00:52:25.137253 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:25.137128    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv" podUID="9905328c-20ac-42d2-a90e-e2190360b082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 00:52:26.140806 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:26.140677    4393 patch_prober.go:29] interesting pod/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 00:52:26.140806 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:26.140739    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv" podUID="9905328c-20ac-42d2-a90e-e2190360b082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 00:52:26.533702 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:52:26.533613    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:52:27.299816 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:27.158573    4393 patch_prober.go:29] interesting pod/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 00:52:27.299816 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:27.158670    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv" podUID="9905328c-20ac-42d2-a90e-e2190360b082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 00:52:28.145403 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:28.135900    4393 patch_prober.go:29] interesting pod/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 00:52:28.145403 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:28.135973    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv" podUID="9905328c-20ac-42d2-a90e-e2190360b082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 00:52:29.135294 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:29.135261    4393 patch_prober.go:29] interesting pod/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 00:52:29.135880 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:29.135321    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv" podUID="9905328c-20ac-42d2-a90e-e2190360b082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 00:52:30.135332 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:30.135293    4393 patch_prober.go:29] interesting pod/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Oct 30 00:52:30.135832 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:30.135354    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv" podUID="9905328c-20ac-42d2-a90e-e2190360b082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Oct 30 00:52:30.581574 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:30.581496    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk"
Oct 30 00:52:31.139504 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:31.139472    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"
Oct 30 00:52:31.139996 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:31.139884    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"
Oct 30 00:52:31.140498 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:31.140475    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"
Oct 30 00:52:34.134053 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:34.134024    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1fa555-54499dd979-2gck7"]
Oct 30 00:52:34.139869 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:34.139839    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fa555-54499dd979-2gck7"
Oct 30 00:52:34.156964 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:34.156938    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1fa555-54499dd979-2gck7"]
Oct 30 00:52:34.162950 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:34.162908    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-1fa555\""
Oct 30 00:52:34.163074 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:34.163011    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-1fa555-dockercfg-85xf4\""
Oct 30 00:52:34.302028 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:34.301990    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t652\" (UniqueName: \"kubernetes.io/projected/81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3-kube-api-access-6t652\") pod \"payload-pre-processing-e2e-api-a-1fa555-54499dd979-2gck7\" (UID: \"81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fa555-54499dd979-2gck7"
Oct 30 00:52:34.374485 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:34.374449    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1fa555-54499dd979-2gck7\" (UID: \"81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fa555-54499dd979-2gck7"
Oct 30 00:52:34.425007 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:34.424923    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1fa555-7576b49db4-bh5h6"]
Oct 30 00:52:34.440621 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:34.440568    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1fa555-7576b49db4-bh5h6"
Oct 30 00:52:34.522454 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:34.520277    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t652\" (UniqueName: \"kubernetes.io/projected/81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3-kube-api-access-6t652\") pod \"payload-pre-processing-e2e-api-a-1fa555-54499dd979-2gck7\" (UID: \"81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fa555-54499dd979-2gck7"
Oct 30 00:52:34.522454 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:34.520333    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1fa555-54499dd979-2gck7\" (UID: \"81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fa555-54499dd979-2gck7"
Oct 30 00:52:34.522454 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:34.521315    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1fa555-7576b49db4-bh5h6"]
Oct 30 00:52:34.522454 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:34.521428    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1fa555-54499dd979-2gck7\" (UID: \"81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fa555-54499dd979-2gck7"
Oct 30 00:52:34.572804 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:34.566360    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t652\" (UniqueName: \"kubernetes.io/projected/81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3-kube-api-access-6t652\") pod \"payload-pre-processing-e2e-api-a-1fa555-54499dd979-2gck7\" (UID: \"81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fa555-54499dd979-2gck7"
Oct 30 00:52:34.628552 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:34.628510    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2n8l\" (UniqueName: \"kubernetes.io/projected/9d646802-1521-40c0-8f2e-b09016feb486-kube-api-access-p2n8l\") pod \"payload-processing-e2e-api-a-1fa555-7576b49db4-bh5h6\" (UID: \"9d646802-1521-40c0-8f2e-b09016feb486\") " pod="openshift-ingress/payload-processing-e2e-api-a-1fa555-7576b49db4-bh5h6"
Oct 30 00:52:34.630026 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:34.629944    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d646802-1521-40c0-8f2e-b09016feb486-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1fa555-7576b49db4-bh5h6\" (UID: \"9d646802-1521-40c0-8f2e-b09016feb486\") " pod="openshift-ingress/payload-processing-e2e-api-a-1fa555-7576b49db4-bh5h6"
Oct 30 00:52:34.731725 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:34.731643    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d646802-1521-40c0-8f2e-b09016feb486-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1fa555-7576b49db4-bh5h6\" (UID: \"9d646802-1521-40c0-8f2e-b09016feb486\") " pod="openshift-ingress/payload-processing-e2e-api-a-1fa555-7576b49db4-bh5h6"
Oct 30 00:52:34.731972 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:34.731957    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2n8l\" (UniqueName: \"kubernetes.io/projected/9d646802-1521-40c0-8f2e-b09016feb486-kube-api-access-p2n8l\") pod \"payload-processing-e2e-api-a-1fa555-7576b49db4-bh5h6\" (UID: \"9d646802-1521-40c0-8f2e-b09016feb486\") " pod="openshift-ingress/payload-processing-e2e-api-a-1fa555-7576b49db4-bh5h6"
Oct 30 00:52:34.732552 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:34.732527    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d646802-1521-40c0-8f2e-b09016feb486-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1fa555-7576b49db4-bh5h6\" (UID: \"9d646802-1521-40c0-8f2e-b09016feb486\") " pod="openshift-ingress/payload-processing-e2e-api-a-1fa555-7576b49db4-bh5h6"
Oct 30 00:52:34.754738 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:34.754699    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2n8l\" (UniqueName: \"kubernetes.io/projected/9d646802-1521-40c0-8f2e-b09016feb486-kube-api-access-p2n8l\") pod \"payload-processing-e2e-api-a-1fa555-7576b49db4-bh5h6\" (UID: \"9d646802-1521-40c0-8f2e-b09016feb486\") " pod="openshift-ingress/payload-processing-e2e-api-a-1fa555-7576b49db4-bh5h6"
Oct 30 00:52:34.756414 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:34.756396    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fa555-54499dd979-2gck7"
Oct 30 00:52:34.863869 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:34.863835    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1fa555-7576b49db4-bh5h6"
Oct 30 00:52:35.065900 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:52:35.065872    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81e86a2b_f8fa_4cf2_b9a5_66f3eacf7fc3.slice/crio-86e6a32298da0828bbdde1bf3bd0dd0cb17876dba448e04b783be5ba50086b0c WatchSource:0}: Error finding container 86e6a32298da0828bbdde1bf3bd0dd0cb17876dba448e04b783be5ba50086b0c: Status 404 returned error can't find the container with id 86e6a32298da0828bbdde1bf3bd0dd0cb17876dba448e04b783be5ba50086b0c
Oct 30 00:52:35.066892 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:35.066764    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1fa555-54499dd979-2gck7"]
Oct 30 00:52:35.187586 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:52:35.187560    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d646802_1521_40c0_8f2e_b09016feb486.slice/crio-40aacc91c23bb6fbc8a67bbcbbd7837d93a41b710b012a3fe1ebde457822b91d WatchSource:0}: Error finding container 40aacc91c23bb6fbc8a67bbcbbd7837d93a41b710b012a3fe1ebde457822b91d: Status 404 returned error can't find the container with id 40aacc91c23bb6fbc8a67bbcbbd7837d93a41b710b012a3fe1ebde457822b91d
Oct 30 00:52:35.190676 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:35.190648    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1fa555-7576b49db4-bh5h6"]
Oct 30 00:52:35.729396 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:35.729361    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1fa555-7576b49db4-bh5h6" event={"ID":"9d646802-1521-40c0-8f2e-b09016feb486","Type":"ContainerStarted","Data":"19275aeaf09b4d3ddd769039bef5c26c8d2f8c2b1dcee884954b89462874c5fa"}
Oct 30 00:52:35.729586 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:35.729407    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1fa555-7576b49db4-bh5h6" event={"ID":"9d646802-1521-40c0-8f2e-b09016feb486","Type":"ContainerStarted","Data":"40aacc91c23bb6fbc8a67bbcbbd7837d93a41b710b012a3fe1ebde457822b91d"}
Oct 30 00:52:35.729649 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:35.729596    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-1fa555-7576b49db4-bh5h6"
Oct 30 00:52:35.731362 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:35.731329    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fa555-54499dd979-2gck7" event={"ID":"81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3","Type":"ContainerStarted","Data":"b5a00da8aeecbf11d51c540967cb13d521b5a65e428c740d0448fd4907cd9929"}
Oct 30 00:52:35.731480 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:35.731381    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fa555-54499dd979-2gck7"
Oct 30 00:52:35.731480 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:35.731401    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fa555-54499dd979-2gck7" event={"ID":"81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3","Type":"ContainerStarted","Data":"86e6a32298da0828bbdde1bf3bd0dd0cb17876dba448e04b783be5ba50086b0c"}
Oct 30 00:52:35.770672 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:35.770624    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-1fa555-7576b49db4-bh5h6" podStartSLOduration=1.770601512 podStartE2EDuration="1.770601512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:35.768227511 +0000 UTC m=+1246.155903900" watchObservedRunningTime="2026-10-30 00:52:35.770601512 +0000 UTC m=+1246.158277900"
Oct 30 00:52:35.811563 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:35.811514    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fa555-54499dd979-2gck7" podStartSLOduration=1.81150171 podStartE2EDuration="1.81150171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:35.806466063 +0000 UTC m=+1246.194142452" watchObservedRunningTime="2026-10-30 00:52:35.81150171 +0000 UTC m=+1246.199178098"
Oct 30 00:52:46.743124 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:46.743093    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fa555-54499dd979-2gck7"
Oct 30 00:52:46.743509 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:46.743160    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-1fa555-7576b49db4-bh5h6"
Oct 30 00:52:53.816808 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:53.816765    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1fa555-d4565947-nh89k"]
Oct 30 00:52:53.822251 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:53.822231    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fa555-d4565947-nh89k"
Oct 30 00:52:53.844237 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:53.844208    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1fa555-d4565947-nh89k"]
Oct 30 00:52:53.845515 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:53.845489    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-1fa555-serving-cert\""
Oct 30 00:52:53.891114 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:53.891077    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ef7dab9-ca94-4f57-85e1-cd0492b23e2b-metrics-certs\") pod \"maas-api-e2e-api-a-1fa555-d4565947-nh89k\" (UID: \"3ef7dab9-ca94-4f57-85e1-cd0492b23e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fa555-d4565947-nh89k"
Oct 30 00:52:53.891338 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:53.891151    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ef7dab9-ca94-4f57-85e1-cd0492b23e2b-openshift-service-ca\") pod \"maas-api-e2e-api-a-1fa555-d4565947-nh89k\" (UID: \"3ef7dab9-ca94-4f57-85e1-cd0492b23e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fa555-d4565947-nh89k"
Oct 30 00:52:53.891338 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:53.891270    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xp9k\" (UniqueName: \"kubernetes.io/projected/3ef7dab9-ca94-4f57-85e1-cd0492b23e2b-kube-api-access-9xp9k\") pod \"maas-api-e2e-api-a-1fa555-d4565947-nh89k\" (UID: \"3ef7dab9-ca94-4f57-85e1-cd0492b23e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fa555-d4565947-nh89k"
Oct 30 00:52:53.891338 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:53.891308    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ef7dab9-ca94-4f57-85e1-cd0492b23e2b-maas-api-tls\") pod \"maas-api-e2e-api-a-1fa555-d4565947-nh89k\" (UID: \"3ef7dab9-ca94-4f57-85e1-cd0492b23e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fa555-d4565947-nh89k"
Oct 30 00:52:54.022307 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:54.022264    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ef7dab9-ca94-4f57-85e1-cd0492b23e2b-openshift-service-ca\") pod \"maas-api-e2e-api-a-1fa555-d4565947-nh89k\" (UID: \"3ef7dab9-ca94-4f57-85e1-cd0492b23e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fa555-d4565947-nh89k"
Oct 30 00:52:54.027805 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:54.023874    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xp9k\" (UniqueName: \"kubernetes.io/projected/3ef7dab9-ca94-4f57-85e1-cd0492b23e2b-kube-api-access-9xp9k\") pod \"maas-api-e2e-api-a-1fa555-d4565947-nh89k\" (UID: \"3ef7dab9-ca94-4f57-85e1-cd0492b23e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fa555-d4565947-nh89k"
Oct 30 00:52:54.046520 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:54.032954    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ef7dab9-ca94-4f57-85e1-cd0492b23e2b-maas-api-tls\") pod \"maas-api-e2e-api-a-1fa555-d4565947-nh89k\" (UID: \"3ef7dab9-ca94-4f57-85e1-cd0492b23e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fa555-d4565947-nh89k"
Oct 30 00:52:54.047473 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:54.047437    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ef7dab9-ca94-4f57-85e1-cd0492b23e2b-metrics-certs\") pod \"maas-api-e2e-api-a-1fa555-d4565947-nh89k\" (UID: \"3ef7dab9-ca94-4f57-85e1-cd0492b23e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fa555-d4565947-nh89k"
Oct 30 00:52:54.077184 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:54.023540    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ef7dab9-ca94-4f57-85e1-cd0492b23e2b-openshift-service-ca\") pod \"maas-api-e2e-api-a-1fa555-d4565947-nh89k\" (UID: \"3ef7dab9-ca94-4f57-85e1-cd0492b23e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fa555-d4565947-nh89k"
Oct 30 00:52:54.077429 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:54.061499    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xp9k\" (UniqueName: \"kubernetes.io/projected/3ef7dab9-ca94-4f57-85e1-cd0492b23e2b-kube-api-access-9xp9k\") pod \"maas-api-e2e-api-a-1fa555-d4565947-nh89k\" (UID: \"3ef7dab9-ca94-4f57-85e1-cd0492b23e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fa555-d4565947-nh89k"
Oct 30 00:52:54.085805 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:54.081260    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ef7dab9-ca94-4f57-85e1-cd0492b23e2b-metrics-certs\") pod \"maas-api-e2e-api-a-1fa555-d4565947-nh89k\" (UID: \"3ef7dab9-ca94-4f57-85e1-cd0492b23e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fa555-d4565947-nh89k"
Oct 30 00:52:54.085805 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:54.082771    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ef7dab9-ca94-4f57-85e1-cd0492b23e2b-maas-api-tls\") pod \"maas-api-e2e-api-a-1fa555-d4565947-nh89k\" (UID: \"3ef7dab9-ca94-4f57-85e1-cd0492b23e2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fa555-d4565947-nh89k"
Oct 30 00:52:54.140146 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:54.140113    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fa555-d4565947-nh89k"
Oct 30 00:52:54.474162 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:52:54.474126    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef7dab9_ca94_4f57_85e1_cd0492b23e2b.slice/crio-1f987e95a0d402c5be21a091d07f497af6a341a05f13300d20abf1ae7f3699d9 WatchSource:0}: Error finding container 1f987e95a0d402c5be21a091d07f497af6a341a05f13300d20abf1ae7f3699d9: Status 404 returned error can't find the container with id 1f987e95a0d402c5be21a091d07f497af6a341a05f13300d20abf1ae7f3699d9
Oct 30 00:52:54.474732 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:54.474655    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1fa555-d4565947-nh89k"]
Oct 30 00:52:55.088880 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:55.083698    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fa555-d4565947-nh89k" event={"ID":"3ef7dab9-ca94-4f57-85e1-cd0492b23e2b","Type":"ContainerStarted","Data":"6dd98110dd39a503696d24243912fd9ae777feb4b9bb0711240c6230494e4a24"}
Oct 30 00:52:55.088880 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:55.083769    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fa555-d4565947-nh89k" event={"ID":"3ef7dab9-ca94-4f57-85e1-cd0492b23e2b","Type":"ContainerStarted","Data":"1f987e95a0d402c5be21a091d07f497af6a341a05f13300d20abf1ae7f3699d9"}
Oct 30 00:52:55.088880 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:55.084870    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fa555-d4565947-nh89k"
Oct 30 00:52:55.152300 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:52:55.152126    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fa555-d4565947-nh89k" podStartSLOduration=1.820213953 podStartE2EDuration="2.15210418s" totalImagesPullingTime="331.890227ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:52:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:52:55.140958251 +0000 UTC m=+1265.528634641" watchObservedRunningTime="2026-10-30 00:52:55.15210418 +0000 UTC m=+1265.539780570"
Oct 30 00:52:56.512847 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:52:56.512804    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:53:02.106825 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:02.106770    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fa555-d4565947-nh89k"
Oct 30 00:53:26.514923 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:53:26.514877    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:53:32.328597 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:32.313210    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d788ba-7b5d769864-b98xr"]
Oct 30 00:53:32.344958 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:32.344921    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d788ba-7b5d769864-b98xr"
Oct 30 00:53:32.345152 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:32.345118    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d788ba-7b5d769864-b98xr"]
Oct 30 00:53:32.375519 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:32.374469    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d788ba-serving-cert\""
Oct 30 00:53:32.388083 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:32.388049    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57d02d2c-140f-4463-a8b5-0fdc4dbabcf8-maas-api-tls\") pod \"maas-api-e2e-api-a-d788ba-7b5d769864-b98xr\" (UID: \"57d02d2c-140f-4463-a8b5-0fdc4dbabcf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d788ba-7b5d769864-b98xr"
Oct 30 00:53:32.388437 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:32.388417    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp5mw\" (UniqueName: \"kubernetes.io/projected/57d02d2c-140f-4463-a8b5-0fdc4dbabcf8-kube-api-access-wp5mw\") pod \"maas-api-e2e-api-a-d788ba-7b5d769864-b98xr\" (UID: \"57d02d2c-140f-4463-a8b5-0fdc4dbabcf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d788ba-7b5d769864-b98xr"
Oct 30 00:53:32.388712 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:32.388698    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57d02d2c-140f-4463-a8b5-0fdc4dbabcf8-metrics-certs\") pod \"maas-api-e2e-api-a-d788ba-7b5d769864-b98xr\" (UID: \"57d02d2c-140f-4463-a8b5-0fdc4dbabcf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d788ba-7b5d769864-b98xr"
Oct 30 00:53:32.388984 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:32.388964    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57d02d2c-140f-4463-a8b5-0fdc4dbabcf8-openshift-service-ca\") pod \"maas-api-e2e-api-a-d788ba-7b5d769864-b98xr\" (UID: \"57d02d2c-140f-4463-a8b5-0fdc4dbabcf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d788ba-7b5d769864-b98xr"
Oct 30 00:53:32.490217 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:32.490181    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp5mw\" (UniqueName: \"kubernetes.io/projected/57d02d2c-140f-4463-a8b5-0fdc4dbabcf8-kube-api-access-wp5mw\") pod \"maas-api-e2e-api-a-d788ba-7b5d769864-b98xr\" (UID: \"57d02d2c-140f-4463-a8b5-0fdc4dbabcf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d788ba-7b5d769864-b98xr"
Oct 30 00:53:32.490388 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:32.490366    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57d02d2c-140f-4463-a8b5-0fdc4dbabcf8-metrics-certs\") pod \"maas-api-e2e-api-a-d788ba-7b5d769864-b98xr\" (UID: \"57d02d2c-140f-4463-a8b5-0fdc4dbabcf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d788ba-7b5d769864-b98xr"
Oct 30 00:53:32.490495 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:32.490478    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57d02d2c-140f-4463-a8b5-0fdc4dbabcf8-openshift-service-ca\") pod \"maas-api-e2e-api-a-d788ba-7b5d769864-b98xr\" (UID: \"57d02d2c-140f-4463-a8b5-0fdc4dbabcf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d788ba-7b5d769864-b98xr"
Oct 30 00:53:32.490599 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:32.490581    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57d02d2c-140f-4463-a8b5-0fdc4dbabcf8-maas-api-tls\") pod \"maas-api-e2e-api-a-d788ba-7b5d769864-b98xr\" (UID: \"57d02d2c-140f-4463-a8b5-0fdc4dbabcf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d788ba-7b5d769864-b98xr"
Oct 30 00:53:32.490730 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:53:32.490714    4393 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-d788ba-serving-cert: secret "maas-api-e2e-api-a-d788ba-serving-cert" not found
Oct 30 00:53:32.490806 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:53:32.490791    4393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57d02d2c-140f-4463-a8b5-0fdc4dbabcf8-maas-api-tls podName:57d02d2c-140f-4463-a8b5-0fdc4dbabcf8 nodeName:}" failed. No retries permitted until 2026-10-30 00:53:32.990758699 +0000 UTC m=+1303.378435067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/57d02d2c-140f-4463-a8b5-0fdc4dbabcf8-maas-api-tls") pod "maas-api-e2e-api-a-d788ba-7b5d769864-b98xr" (UID: "57d02d2c-140f-4463-a8b5-0fdc4dbabcf8") : secret "maas-api-e2e-api-a-d788ba-serving-cert" not found
Oct 30 00:53:32.491239 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:32.491214    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57d02d2c-140f-4463-a8b5-0fdc4dbabcf8-openshift-service-ca\") pod \"maas-api-e2e-api-a-d788ba-7b5d769864-b98xr\" (UID: \"57d02d2c-140f-4463-a8b5-0fdc4dbabcf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d788ba-7b5d769864-b98xr"
Oct 30 00:53:32.493145 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:32.493123    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57d02d2c-140f-4463-a8b5-0fdc4dbabcf8-metrics-certs\") pod \"maas-api-e2e-api-a-d788ba-7b5d769864-b98xr\" (UID: \"57d02d2c-140f-4463-a8b5-0fdc4dbabcf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d788ba-7b5d769864-b98xr"
Oct 30 00:53:32.513582 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:32.513557    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp5mw\" (UniqueName: \"kubernetes.io/projected/57d02d2c-140f-4463-a8b5-0fdc4dbabcf8-kube-api-access-wp5mw\") pod \"maas-api-e2e-api-a-d788ba-7b5d769864-b98xr\" (UID: \"57d02d2c-140f-4463-a8b5-0fdc4dbabcf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d788ba-7b5d769864-b98xr"
Oct 30 00:53:33.026805 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:33.017678    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57d02d2c-140f-4463-a8b5-0fdc4dbabcf8-maas-api-tls\") pod \"maas-api-e2e-api-a-d788ba-7b5d769864-b98xr\" (UID: \"57d02d2c-140f-4463-a8b5-0fdc4dbabcf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d788ba-7b5d769864-b98xr"
Oct 30 00:53:33.076623 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:33.076588    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57d02d2c-140f-4463-a8b5-0fdc4dbabcf8-maas-api-tls\") pod \"maas-api-e2e-api-a-d788ba-7b5d769864-b98xr\" (UID: \"57d02d2c-140f-4463-a8b5-0fdc4dbabcf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d788ba-7b5d769864-b98xr"
Oct 30 00:53:33.300077 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:33.299501    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d788ba-7b5d769864-b98xr"
Oct 30 00:53:33.710020 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:33.709920    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d788ba-7b5d769864-b98xr"]
Oct 30 00:53:33.921885 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:33.917424    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d788ba-7b5d769864-b98xr" event={"ID":"57d02d2c-140f-4463-a8b5-0fdc4dbabcf8","Type":"ContainerStarted","Data":"c93141e082a3447ccc5e2b5ac65af815a94134d30a9f02fe90b0d8f864b54a9b"}
Oct 30 00:53:34.974577 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:34.974533    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d788ba-7b5d769864-b98xr" event={"ID":"57d02d2c-140f-4463-a8b5-0fdc4dbabcf8","Type":"ContainerStarted","Data":"85d9506af8a0ebdcb7f828481960018f8fceaf635d1d81f0ee9dff13d6379238"}
Oct 30 00:53:34.985372 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:34.976394    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d788ba-7b5d769864-b98xr"
Oct 30 00:53:35.048805 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:35.047945    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d788ba-7b5d769864-b98xr" podStartSLOduration=2.558386321 podStartE2EDuration="3.047924888s" totalImagesPullingTime="489.538567ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:35.029164889 +0000 UTC m=+1305.416841305" watchObservedRunningTime="2026-10-30 00:53:35.047924888 +0000 UTC m=+1305.435601279"
Oct 30 00:53:36.287656 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:36.287614    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-q78mf"]
Oct 30 00:53:41.291181 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:41.291110    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-q78mf" podUID="999e3533-c8ce-4fff-a51d-1e7938955602" containerName="payload-pre-processing" containerID="cri-o://d4e0036339990c98261d3900f31a6ba033b706333d369cc902998548ff2531e0" gracePeriod=25
Oct 30 00:53:42.044813 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:42.044768    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d788ba-7b5d769864-b98xr"
Oct 30 00:53:44.535805 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:44.535286    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-q78mf" podUID="999e3533-c8ce-4fff-a51d-1e7938955602" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 00:53:49.302172 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:49.302133    4393 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-56.ec2.internal" podUID="21539217511a9b702eb77d59ab0b339d" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:53.010031 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:53.009995    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd"]
Oct 30 00:53:53.015512 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:53.015488    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd"
Oct 30 00:53:53.056874 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:53.043892    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-bdc256-serving-cert\""
Oct 30 00:53:53.056874 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:53.052488    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd"]
Oct 30 00:53:53.109714 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:53.109671    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmx8h\" (UniqueName: \"kubernetes.io/projected/af96c512-8935-4b9d-86b6-18c374aeb0a7-kube-api-access-lmx8h\") pod \"maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd\" (UID: \"af96c512-8935-4b9d-86b6-18c374aeb0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd"
Oct 30 00:53:53.110976 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:53.110936    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af96c512-8935-4b9d-86b6-18c374aeb0a7-maas-api-tls\") pod \"maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd\" (UID: \"af96c512-8935-4b9d-86b6-18c374aeb0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd"
Oct 30 00:53:53.111238 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:53.111218    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af96c512-8935-4b9d-86b6-18c374aeb0a7-openshift-service-ca\") pod \"maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd\" (UID: \"af96c512-8935-4b9d-86b6-18c374aeb0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd"
Oct 30 00:53:53.112133 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:53.112096    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af96c512-8935-4b9d-86b6-18c374aeb0a7-metrics-certs\") pod \"maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd\" (UID: \"af96c512-8935-4b9d-86b6-18c374aeb0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd"
Oct 30 00:53:53.238807 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:53.236938    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af96c512-8935-4b9d-86b6-18c374aeb0a7-metrics-certs\") pod \"maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd\" (UID: \"af96c512-8935-4b9d-86b6-18c374aeb0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd"
Oct 30 00:53:53.275861 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:53.275620    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmx8h\" (UniqueName: \"kubernetes.io/projected/af96c512-8935-4b9d-86b6-18c374aeb0a7-kube-api-access-lmx8h\") pod \"maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd\" (UID: \"af96c512-8935-4b9d-86b6-18c374aeb0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd"
Oct 30 00:53:53.275861 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:53.275676    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af96c512-8935-4b9d-86b6-18c374aeb0a7-maas-api-tls\") pod \"maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd\" (UID: \"af96c512-8935-4b9d-86b6-18c374aeb0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd"
Oct 30 00:53:53.275861 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:53.275705    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af96c512-8935-4b9d-86b6-18c374aeb0a7-openshift-service-ca\") pod \"maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd\" (UID: \"af96c512-8935-4b9d-86b6-18c374aeb0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd"
Oct 30 00:53:53.284815 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:53.282845    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af96c512-8935-4b9d-86b6-18c374aeb0a7-openshift-service-ca\") pod \"maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd\" (UID: \"af96c512-8935-4b9d-86b6-18c374aeb0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd"
Oct 30 00:53:53.284815 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:53:53.283333    4393 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-bdc256-serving-cert: secret "maas-api-e2e-api-b-bdc256-serving-cert" not found
Oct 30 00:53:53.284815 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:53:53.283435    4393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af96c512-8935-4b9d-86b6-18c374aeb0a7-maas-api-tls podName:af96c512-8935-4b9d-86b6-18c374aeb0a7 nodeName:}" failed. No retries permitted until 2026-10-30 00:53:53.783409909 +0000 UTC m=+1324.171086276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/af96c512-8935-4b9d-86b6-18c374aeb0a7-maas-api-tls") pod "maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd" (UID: "af96c512-8935-4b9d-86b6-18c374aeb0a7") : secret "maas-api-e2e-api-b-bdc256-serving-cert" not found
Oct 30 00:53:53.305513 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:53.305479    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af96c512-8935-4b9d-86b6-18c374aeb0a7-metrics-certs\") pod \"maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd\" (UID: \"af96c512-8935-4b9d-86b6-18c374aeb0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd"
Oct 30 00:53:53.322557 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:53.322522    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmx8h\" (UniqueName: \"kubernetes.io/projected/af96c512-8935-4b9d-86b6-18c374aeb0a7-kube-api-access-lmx8h\") pod \"maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd\" (UID: \"af96c512-8935-4b9d-86b6-18c374aeb0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd"
Oct 30 00:53:53.789054 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:53.788645    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af96c512-8935-4b9d-86b6-18c374aeb0a7-maas-api-tls\") pod \"maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd\" (UID: \"af96c512-8935-4b9d-86b6-18c374aeb0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd"
Oct 30 00:53:53.792156 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:53.792125    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af96c512-8935-4b9d-86b6-18c374aeb0a7-maas-api-tls\") pod \"maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd\" (UID: \"af96c512-8935-4b9d-86b6-18c374aeb0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd"
Oct 30 00:53:53.946166 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:53.946090    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd"
Oct 30 00:53:54.647866 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:54.611921    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-q78mf" podUID="999e3533-c8ce-4fff-a51d-1e7938955602" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 00:53:54.900330 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:53:54.900079    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf96c512_8935_4b9d_86b6_18c374aeb0a7.slice/crio-3c2100580c60cde350f93347e8af00c90ab7e21586acaa22332d4ca371dda688 WatchSource:0}: Error finding container 3c2100580c60cde350f93347e8af00c90ab7e21586acaa22332d4ca371dda688: Status 404 returned error can't find the container with id 3c2100580c60cde350f93347e8af00c90ab7e21586acaa22332d4ca371dda688
Oct 30 00:53:54.955027 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:54.951810    4393 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:53:54.974328 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:54.974291    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd"]
Oct 30 00:53:55.592929 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:55.592887    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-kfpcv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:53:55.593117 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:55.592983    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv" podUID="a5b4591e-8713-40c5-b33c-e6505356ee29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:53:55.645802 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:55.644890    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd" event={"ID":"af96c512-8935-4b9d-86b6-18c374aeb0a7","Type":"ContainerStarted","Data":"3c2100580c60cde350f93347e8af00c90ab7e21586acaa22332d4ca371dda688"}
Oct 30 00:53:56.515810 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:53:56.515752    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:53:56.655294 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:56.655258    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd" event={"ID":"af96c512-8935-4b9d-86b6-18c374aeb0a7","Type":"ContainerStarted","Data":"772db66e0f1eb40eb5ee9b3f16a2128273720bb1eeae132cd0aa10a6a6727c03"}
Oct 30 00:53:56.655482 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:56.655325    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd"
Oct 30 00:53:56.696795 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:53:56.696739    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd" podStartSLOduration=4.183894697 podStartE2EDuration="4.696724545s" totalImagesPullingTime="512.829848ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:53:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:53:56.696230899 +0000 UTC m=+1327.083907291" watchObservedRunningTime="2026-10-30 00:53:56.696724545 +0000 UTC m=+1327.084400936"
Oct 30 00:54:01.614868 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:01.614836    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-q78mf"
Oct 30 00:54:01.721029 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:01.720991    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/999e3533-c8ce-4fff-a51d-1e7938955602-kube-api-access-br4zp\" (UniqueName: \"kubernetes.io/projected/999e3533-c8ce-4fff-a51d-1e7938955602-kube-api-access-br4zp\") pod \"999e3533-c8ce-4fff-a51d-1e7938955602\" (UID: \"999e3533-c8ce-4fff-a51d-1e7938955602\") "
Oct 30 00:54:01.721210 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:01.721074    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/999e3533-c8ce-4fff-a51d-1e7938955602-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/999e3533-c8ce-4fff-a51d-1e7938955602-plugins-config-volume\") pod \"999e3533-c8ce-4fff-a51d-1e7938955602\" (UID: \"999e3533-c8ce-4fff-a51d-1e7938955602\") "
Oct 30 00:54:01.721472 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:01.721445    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/999e3533-c8ce-4fff-a51d-1e7938955602-plugins-config-volume" pod "999e3533-c8ce-4fff-a51d-1e7938955602" (UID: "999e3533-c8ce-4fff-a51d-1e7938955602"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:54:01.722021 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:01.721993    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/999e3533-c8ce-4fff-a51d-1e7938955602-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:54:01.723432 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:01.723400    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/999e3533-c8ce-4fff-a51d-1e7938955602-kube-api-access-br4zp" pod "999e3533-c8ce-4fff-a51d-1e7938955602" (UID: "999e3533-c8ce-4fff-a51d-1e7938955602"). InnerVolumeSpecName "kube-api-access-br4zp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:54:01.770220 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:01.770185    4393 generic.go:356] "Generic (PLEG): container finished" podID="999e3533-c8ce-4fff-a51d-1e7938955602" containerID="d4e0036339990c98261d3900f31a6ba033b706333d369cc902998548ff2531e0" exitCode=0
Oct 30 00:54:01.770394 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:01.770266    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-q78mf"
Oct 30 00:54:01.770394 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:01.770385    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-q78mf" event={"ID":"999e3533-c8ce-4fff-a51d-1e7938955602","Type":"ContainerDied","Data":"d4e0036339990c98261d3900f31a6ba033b706333d369cc902998548ff2531e0"}
Oct 30 00:54:01.770510 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:01.770436    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-q78mf" event={"ID":"999e3533-c8ce-4fff-a51d-1e7938955602","Type":"ContainerDied","Data":"0957a88561743dc8eb0c4911522c23a78ff3163533ebfc5fd774fbd33d428065"}
Oct 30 00:54:01.770510 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:01.770467    4393 scope.go:122] "RemoveContainer" containerID="d4e0036339990c98261d3900f31a6ba033b706333d369cc902998548ff2531e0"
Oct 30 00:54:01.783346 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:01.783317    4393 scope.go:122] "RemoveContainer" containerID="d4e0036339990c98261d3900f31a6ba033b706333d369cc902998548ff2531e0"
Oct 30 00:54:01.783702 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:54:01.783665    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4e0036339990c98261d3900f31a6ba033b706333d369cc902998548ff2531e0\": container with ID starting with d4e0036339990c98261d3900f31a6ba033b706333d369cc902998548ff2531e0 not found: ID does not exist" containerID="d4e0036339990c98261d3900f31a6ba033b706333d369cc902998548ff2531e0"
Oct 30 00:54:01.783819 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:01.783705    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4e0036339990c98261d3900f31a6ba033b706333d369cc902998548ff2531e0"} err="failed to get container status \"d4e0036339990c98261d3900f31a6ba033b706333d369cc902998548ff2531e0\": rpc error: code = NotFound desc = could not find container \"d4e0036339990c98261d3900f31a6ba033b706333d369cc902998548ff2531e0\": container with ID starting with d4e0036339990c98261d3900f31a6ba033b706333d369cc902998548ff2531e0 not found: ID does not exist"
Oct 30 00:54:01.823810 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:01.823744    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-q78mf"]
Oct 30 00:54:01.823810 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:01.823768    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br4zp\" (UniqueName: \"kubernetes.io/projected/999e3533-c8ce-4fff-a51d-1e7938955602-kube-api-access-br4zp\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:54:01.828725 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:01.828664    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d7a6db25-67ddd99559-q78mf"]
Oct 30 00:54:02.733562 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:02.733536    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd"
Oct 30 00:54:03.726732 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:03.726703    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="999e3533-c8ce-4fff-a51d-1e7938955602" path="/var/lib/kubelet/pods/999e3533-c8ce-4fff-a51d-1e7938955602/volumes"
Oct 30 00:54:19.496316 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:19.496282    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786"]
Oct 30 00:54:19.497100 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:19.497067    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786" podUID="6f2d0713-1871-487b-857e-f10b16ad2622" containerName="maas-api" containerID="cri-o://db71b10d07acf584e8cf986e47c254affd9e7fc467a13464787c01f80192faf8" gracePeriod=30
Oct 30 00:54:19.861865 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:19.831982    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786" podUID="6f2d0713-1871-487b-857e-f10b16ad2622" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.117:8443/health\": dial tcp 10.133.0.117:8443: connect: connection refused"
Oct 30 00:54:20.293847 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:20.293769    4393 generic.go:356] "Generic (PLEG): container finished" podID="6f2d0713-1871-487b-857e-f10b16ad2622" containerID="db71b10d07acf584e8cf986e47c254affd9e7fc467a13464787c01f80192faf8" exitCode=0
Oct 30 00:54:20.294013 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:20.293955    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786" event={"ID":"6f2d0713-1871-487b-857e-f10b16ad2622","Type":"ContainerDied","Data":"db71b10d07acf584e8cf986e47c254affd9e7fc467a13464787c01f80192faf8"}
Oct 30 00:54:20.331514 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:20.331489    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786"
Oct 30 00:54:20.426562 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:20.426525    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f2d0713-1871-487b-857e-f10b16ad2622-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f2d0713-1871-487b-857e-f10b16ad2622-metrics-certs\") pod \"6f2d0713-1871-487b-857e-f10b16ad2622\" (UID: \"6f2d0713-1871-487b-857e-f10b16ad2622\") "
Oct 30 00:54:20.426761 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:20.426692    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f2d0713-1871-487b-857e-f10b16ad2622-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f2d0713-1871-487b-857e-f10b16ad2622-maas-api-tls\") pod \"6f2d0713-1871-487b-857e-f10b16ad2622\" (UID: \"6f2d0713-1871-487b-857e-f10b16ad2622\") "
Oct 30 00:54:20.426897 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:20.426820    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f2d0713-1871-487b-857e-f10b16ad2622-kube-api-access-xgtfn\" (UniqueName: \"kubernetes.io/projected/6f2d0713-1871-487b-857e-f10b16ad2622-kube-api-access-xgtfn\") pod \"6f2d0713-1871-487b-857e-f10b16ad2622\" (UID: \"6f2d0713-1871-487b-857e-f10b16ad2622\") "
Oct 30 00:54:20.426973 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:20.426954    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f2d0713-1871-487b-857e-f10b16ad2622-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2d0713-1871-487b-857e-f10b16ad2622-openshift-service-ca\") pod \"6f2d0713-1871-487b-857e-f10b16ad2622\" (UID: \"6f2d0713-1871-487b-857e-f10b16ad2622\") "
Oct 30 00:54:20.427481 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:20.427384    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f2d0713-1871-487b-857e-f10b16ad2622-openshift-service-ca" pod "6f2d0713-1871-487b-857e-f10b16ad2622" (UID: "6f2d0713-1871-487b-857e-f10b16ad2622"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:54:20.427699 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:20.427677    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2d0713-1871-487b-857e-f10b16ad2622-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:54:20.429093 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:20.429063    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f2d0713-1871-487b-857e-f10b16ad2622-maas-api-tls" pod "6f2d0713-1871-487b-857e-f10b16ad2622" (UID: "6f2d0713-1871-487b-857e-f10b16ad2622"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:54:20.429437 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:20.429419    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f2d0713-1871-487b-857e-f10b16ad2622-metrics-certs" pod "6f2d0713-1871-487b-857e-f10b16ad2622" (UID: "6f2d0713-1871-487b-857e-f10b16ad2622"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:54:20.429602 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:20.429582    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f2d0713-1871-487b-857e-f10b16ad2622-kube-api-access-xgtfn" pod "6f2d0713-1871-487b-857e-f10b16ad2622" (UID: "6f2d0713-1871-487b-857e-f10b16ad2622"). InnerVolumeSpecName "kube-api-access-xgtfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:54:20.529692 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:20.529663    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgtfn\" (UniqueName: \"kubernetes.io/projected/6f2d0713-1871-487b-857e-f10b16ad2622-kube-api-access-xgtfn\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:54:20.529692 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:20.529689    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f2d0713-1871-487b-857e-f10b16ad2622-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:54:20.529692 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:20.529701    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f2d0713-1871-487b-857e-f10b16ad2622-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:54:21.303991 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:21.303956    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786"
Oct 30 00:54:21.303991 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:21.303986    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786" event={"ID":"6f2d0713-1871-487b-857e-f10b16ad2622","Type":"ContainerDied","Data":"a040b7969a8fc3298d2c3aba549ddd07b6bf892b0ee9f2ccf1c185c44d41bbb6"}
Oct 30 00:54:21.304209 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:21.304052    4393 scope.go:122] "RemoveContainer" containerID="db71b10d07acf584e8cf986e47c254affd9e7fc467a13464787c01f80192faf8"
Oct 30 00:54:21.360514 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:21.360481    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786"]
Oct 30 00:54:21.371555 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:21.371524    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-cw786"]
Oct 30 00:54:21.727976 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:21.727939    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f2d0713-1871-487b-857e-f10b16ad2622" path="/var/lib/kubelet/pods/6f2d0713-1871-487b-857e-f10b16ad2622/volumes"
Oct 30 00:54:26.511653 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:54:26.511613    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:54:27.621076 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:27.621037    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-52785f5e-5b49f86876-8268t"]
Oct 30 00:54:27.621977 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:27.621957    4393 cpu_manager.go:414] "containerMap: removing container" podUID="6f2d0713-1871-487b-857e-f10b16ad2622" containerName="maas-api"
Oct 30 00:54:27.621977 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:27.621978    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f2d0713-1871-487b-857e-f10b16ad2622" containerName="maas-api"
Oct 30 00:54:27.622093 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:27.621993    4393 cpu_manager.go:414] "containerMap: removing container" podUID="999e3533-c8ce-4fff-a51d-1e7938955602" containerName="payload-pre-processing"
Oct 30 00:54:27.622093 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:27.621999    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="999e3533-c8ce-4fff-a51d-1e7938955602" containerName="payload-pre-processing"
Oct 30 00:54:27.622156 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:27.622143    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f2d0713-1871-487b-857e-f10b16ad2622" containerName="maas-api"
Oct 30 00:54:27.622156 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:27.622152    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="999e3533-c8ce-4fff-a51d-1e7938955602" containerName="payload-pre-processing"
Oct 30 00:54:27.627846 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:27.627799    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-52785f5e-5b49f86876-8268t"
Oct 30 00:54:27.647108 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:27.647064    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-52785f5e-5b49f86876-8268t"]
Oct 30 00:54:27.648978 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:27.648940    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-52785f5e-serving-cert\""
Oct 30 00:54:27.811249 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:27.811202    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f559e0b-04b7-48f8-a365-490524c81a47-metrics-certs\") pod \"maas-api-e2e-mt-52785f5e-5b49f86876-8268t\" (UID: \"7f559e0b-04b7-48f8-a365-490524c81a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-52785f5e-5b49f86876-8268t"
Oct 30 00:54:27.821066 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:27.821023    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f559e0b-04b7-48f8-a365-490524c81a47-openshift-service-ca\") pod \"maas-api-e2e-mt-52785f5e-5b49f86876-8268t\" (UID: \"7f559e0b-04b7-48f8-a365-490524c81a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-52785f5e-5b49f86876-8268t"
Oct 30 00:54:27.821433 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:27.821412    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f559e0b-04b7-48f8-a365-490524c81a47-maas-api-tls\") pod \"maas-api-e2e-mt-52785f5e-5b49f86876-8268t\" (UID: \"7f559e0b-04b7-48f8-a365-490524c81a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-52785f5e-5b49f86876-8268t"
Oct 30 00:54:27.840542 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:27.840506    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8p9k\" (UniqueName: \"kubernetes.io/projected/7f559e0b-04b7-48f8-a365-490524c81a47-kube-api-access-s8p9k\") pod \"maas-api-e2e-mt-52785f5e-5b49f86876-8268t\" (UID: \"7f559e0b-04b7-48f8-a365-490524c81a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-52785f5e-5b49f86876-8268t"
Oct 30 00:54:27.942523 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:27.942428    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f559e0b-04b7-48f8-a365-490524c81a47-openshift-service-ca\") pod \"maas-api-e2e-mt-52785f5e-5b49f86876-8268t\" (UID: \"7f559e0b-04b7-48f8-a365-490524c81a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-52785f5e-5b49f86876-8268t"
Oct 30 00:54:27.943745 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:27.943719    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f559e0b-04b7-48f8-a365-490524c81a47-maas-api-tls\") pod \"maas-api-e2e-mt-52785f5e-5b49f86876-8268t\" (UID: \"7f559e0b-04b7-48f8-a365-490524c81a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-52785f5e-5b49f86876-8268t"
Oct 30 00:54:27.944272 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:27.944250    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8p9k\" (UniqueName: \"kubernetes.io/projected/7f559e0b-04b7-48f8-a365-490524c81a47-kube-api-access-s8p9k\") pod \"maas-api-e2e-mt-52785f5e-5b49f86876-8268t\" (UID: \"7f559e0b-04b7-48f8-a365-490524c81a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-52785f5e-5b49f86876-8268t"
Oct 30 00:54:27.950793 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:27.945270    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f559e0b-04b7-48f8-a365-490524c81a47-metrics-certs\") pod \"maas-api-e2e-mt-52785f5e-5b49f86876-8268t\" (UID: \"7f559e0b-04b7-48f8-a365-490524c81a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-52785f5e-5b49f86876-8268t"
Oct 30 00:54:27.951858 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:54:27.943997    4393 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-52785f5e-serving-cert: secret "maas-api-e2e-mt-52785f5e-serving-cert" not found
Oct 30 00:54:27.952078 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:54:27.952066    4393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f559e0b-04b7-48f8-a365-490524c81a47-maas-api-tls podName:7f559e0b-04b7-48f8-a365-490524c81a47 nodeName:}" failed. No retries permitted until 2026-10-30 00:54:28.452039859 +0000 UTC m=+1358.839716231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7f559e0b-04b7-48f8-a365-490524c81a47-maas-api-tls") pod "maas-api-e2e-mt-52785f5e-5b49f86876-8268t" (UID: "7f559e0b-04b7-48f8-a365-490524c81a47") : secret "maas-api-e2e-mt-52785f5e-serving-cert" not found
Oct 30 00:54:27.952375 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:27.943603    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f559e0b-04b7-48f8-a365-490524c81a47-openshift-service-ca\") pod \"maas-api-e2e-mt-52785f5e-5b49f86876-8268t\" (UID: \"7f559e0b-04b7-48f8-a365-490524c81a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-52785f5e-5b49f86876-8268t"
Oct 30 00:54:27.954600 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:27.954570    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f559e0b-04b7-48f8-a365-490524c81a47-metrics-certs\") pod \"maas-api-e2e-mt-52785f5e-5b49f86876-8268t\" (UID: \"7f559e0b-04b7-48f8-a365-490524c81a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-52785f5e-5b49f86876-8268t"
Oct 30 00:54:27.967485 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:27.967456    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8p9k\" (UniqueName: \"kubernetes.io/projected/7f559e0b-04b7-48f8-a365-490524c81a47-kube-api-access-s8p9k\") pod \"maas-api-e2e-mt-52785f5e-5b49f86876-8268t\" (UID: \"7f559e0b-04b7-48f8-a365-490524c81a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-52785f5e-5b49f86876-8268t"
Oct 30 00:54:28.458897 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:28.458864    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f559e0b-04b7-48f8-a365-490524c81a47-maas-api-tls\") pod \"maas-api-e2e-mt-52785f5e-5b49f86876-8268t\" (UID: \"7f559e0b-04b7-48f8-a365-490524c81a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-52785f5e-5b49f86876-8268t"
Oct 30 00:54:28.461422 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:28.461391    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f559e0b-04b7-48f8-a365-490524c81a47-maas-api-tls\") pod \"maas-api-e2e-mt-52785f5e-5b49f86876-8268t\" (UID: \"7f559e0b-04b7-48f8-a365-490524c81a47\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-52785f5e-5b49f86876-8268t"
Oct 30 00:54:28.548882 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:28.548853    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-52785f5e-5b49f86876-8268t"
Oct 30 00:54:28.859964 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:54:28.859929    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f559e0b_04b7_48f8_a365_490524c81a47.slice/crio-326334d9fda56f705bc39e2221807ea55a79d242ff6027f2645d265c5e4f7816 WatchSource:0}: Error finding container 326334d9fda56f705bc39e2221807ea55a79d242ff6027f2645d265c5e4f7816: Status 404 returned error can't find the container with id 326334d9fda56f705bc39e2221807ea55a79d242ff6027f2645d265c5e4f7816
Oct 30 00:54:28.860925 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:28.860903    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-52785f5e-5b49f86876-8268t"]
Oct 30 00:54:29.834379 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:29.834344    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-52785f5e-5b49f86876-8268t" event={"ID":"7f559e0b-04b7-48f8-a365-490524c81a47","Type":"ContainerStarted","Data":"4040c0119f4b58dbb75e4125a9fa3c48fe5a39912d88fbee8a037f38caa6a4b2"}
Oct 30 00:54:29.834567 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:29.834395    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-52785f5e-5b49f86876-8268t" event={"ID":"7f559e0b-04b7-48f8-a365-490524c81a47","Type":"ContainerStarted","Data":"326334d9fda56f705bc39e2221807ea55a79d242ff6027f2645d265c5e4f7816"}
Oct 30 00:54:29.834567 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:29.834415    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-6fpr2"]
Oct 30 00:54:29.839477 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:29.839451    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-6fpr2"]
Oct 30 00:54:29.839585 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:29.839491    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-52785f5e-5b49f86876-8268t"
Oct 30 00:54:29.839585 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:29.839555    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-6fpr2"
Oct 30 00:54:29.858136 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:29.858110    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d7a6db25-dockercfg-4fgll\""
Oct 30 00:54:29.858136 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:29.858126    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d7a6db25\""
Oct 30 00:54:29.876477 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:29.876402    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-52785f5e-5b49f86876-8268t" podStartSLOduration=2.596530018 podStartE2EDuration="2.876379309s" totalImagesPullingTime="279.849291ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:29.873231385 +0000 UTC m=+1360.260907789" watchObservedRunningTime="2026-10-30 00:54:29.876379309 +0000 UTC m=+1360.264055702"
Oct 30 00:54:29.913157 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:29.913114    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0ffbbc-535f-428b-a078-ddde09364f7e-plugins-config-volume\") pod \"payload-processing-e2e-mt-d7a6db25-56b6fb9474-6fpr2\" (UID: \"6f0ffbbc-535f-428b-a078-ddde09364f7e\") " pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-6fpr2"
Oct 30 00:54:29.913447 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:29.913417    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrn8c\" (UniqueName: \"kubernetes.io/projected/6f0ffbbc-535f-428b-a078-ddde09364f7e-kube-api-access-wrn8c\") pod \"payload-processing-e2e-mt-d7a6db25-56b6fb9474-6fpr2\" (UID: \"6f0ffbbc-535f-428b-a078-ddde09364f7e\") " pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-6fpr2"
Oct 30 00:54:30.023189 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:30.023156    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0ffbbc-535f-428b-a078-ddde09364f7e-plugins-config-volume\") pod \"payload-processing-e2e-mt-d7a6db25-56b6fb9474-6fpr2\" (UID: \"6f0ffbbc-535f-428b-a078-ddde09364f7e\") " pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-6fpr2"
Oct 30 00:54:30.040928 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:30.030191    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0ffbbc-535f-428b-a078-ddde09364f7e-plugins-config-volume\") pod \"payload-processing-e2e-mt-d7a6db25-56b6fb9474-6fpr2\" (UID: \"6f0ffbbc-535f-428b-a078-ddde09364f7e\") " pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-6fpr2"
Oct 30 00:54:30.041256 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:30.041233    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrn8c\" (UniqueName: \"kubernetes.io/projected/6f0ffbbc-535f-428b-a078-ddde09364f7e-kube-api-access-wrn8c\") pod \"payload-processing-e2e-mt-d7a6db25-56b6fb9474-6fpr2\" (UID: \"6f0ffbbc-535f-428b-a078-ddde09364f7e\") " pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-6fpr2"
Oct 30 00:54:30.092258 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:30.092147    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrn8c\" (UniqueName: \"kubernetes.io/projected/6f0ffbbc-535f-428b-a078-ddde09364f7e-kube-api-access-wrn8c\") pod \"payload-processing-e2e-mt-d7a6db25-56b6fb9474-6fpr2\" (UID: \"6f0ffbbc-535f-428b-a078-ddde09364f7e\") " pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-6fpr2"
Oct 30 00:54:30.157583 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:30.157502    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-6fpr2"
Oct 30 00:54:30.607401 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:54:30.607367    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0ffbbc_535f_428b_a078_ddde09364f7e.slice/crio-e4a0c876e46b5f1170deef87f0eef036ff02ee040b915d220b376beca60ac5f4 WatchSource:0}: Error finding container e4a0c876e46b5f1170deef87f0eef036ff02ee040b915d220b376beca60ac5f4: Status 404 returned error can't find the container with id e4a0c876e46b5f1170deef87f0eef036ff02ee040b915d220b376beca60ac5f4
Oct 30 00:54:30.609675 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:30.609651    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-6fpr2"]
Oct 30 00:54:30.841384 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:30.841341    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-6fpr2" event={"ID":"6f0ffbbc-535f-428b-a078-ddde09364f7e","Type":"ContainerStarted","Data":"8617253c737044aa5cad9c5e4acb24cb90cab2e903616528e33295f04b355d55"}
Oct 30 00:54:30.841563 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:30.841404    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-6fpr2" event={"ID":"6f0ffbbc-535f-428b-a078-ddde09364f7e","Type":"ContainerStarted","Data":"e4a0c876e46b5f1170deef87f0eef036ff02ee040b915d220b376beca60ac5f4"}
Oct 30 00:54:30.841638 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:30.841619    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-6fpr2"
Oct 30 00:54:30.885087 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:30.885038    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-6fpr2" podStartSLOduration=1.885022805 podStartE2EDuration="1.885022805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:30.879246709 +0000 UTC m=+1361.266923101" watchObservedRunningTime="2026-10-30 00:54:30.885022805 +0000 UTC m=+1361.272699193"
Oct 30 00:54:36.442528 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:36.442485    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"]
Oct 30 00:54:36.448057 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:36.448035    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"
Oct 30 00:54:36.484766 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:36.484675    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"]
Oct 30 00:54:36.502756 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:36.502721    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e50cc2ce-0447-448d-9c9f-b93131d3fcba-workload-certs\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn\" (UID: \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"
Oct 30 00:54:36.502756 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:36.502772    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e50cc2ce-0447-448d-9c9f-b93131d3fcba-istio-podinfo\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn\" (UID: \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"
Oct 30 00:54:36.503062 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:36.502945    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e50cc2ce-0447-448d-9c9f-b93131d3fcba-istio-token\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn\" (UID: \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"
Oct 30 00:54:36.503062 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:36.503005    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e50cc2ce-0447-448d-9c9f-b93131d3fcba-credential-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn\" (UID: \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"
Oct 30 00:54:36.503179 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:36.503106    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e50cc2ce-0447-448d-9c9f-b93131d3fcba-istio-data\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn\" (UID: \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"
Oct 30 00:54:36.503179 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:36.503139    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e50cc2ce-0447-448d-9c9f-b93131d3fcba-workload-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn\" (UID: \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"
Oct 30 00:54:36.503296 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:36.503218    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e50cc2ce-0447-448d-9c9f-b93131d3fcba-istio-envoy\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn\" (UID: \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"
Oct 30 00:54:36.503296 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:36.503285    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpkwq\" (UniqueName: \"kubernetes.io/projected/e50cc2ce-0447-448d-9c9f-b93131d3fcba-kube-api-access-qpkwq\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn\" (UID: \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"
Oct 30 00:54:36.503443 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:36.503424    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e50cc2ce-0447-448d-9c9f-b93131d3fcba-istiod-ca-cert\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn\" (UID: \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"
Oct 30 00:54:36.604484 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:36.604452    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e50cc2ce-0447-448d-9c9f-b93131d3fcba-workload-certs\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn\" (UID: \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"
Oct 30 00:54:36.604654 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:36.604504    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e50cc2ce-0447-448d-9c9f-b93131d3fcba-istio-podinfo\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn\" (UID: \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"
Oct 30 00:54:36.604654 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:36.604554    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e50cc2ce-0447-448d-9c9f-b93131d3fcba-istio-token\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn\" (UID: \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"
Oct 30 00:54:36.604654 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:36.604584    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e50cc2ce-0447-448d-9c9f-b93131d3fcba-credential-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn\" (UID: \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"
Oct 30 00:54:36.604794 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:36.604656    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e50cc2ce-0447-448d-9c9f-b93131d3fcba-istio-data\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn\" (UID: \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"
Oct 30 00:54:36.604794 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:36.604686    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e50cc2ce-0447-448d-9c9f-b93131d3fcba-workload-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn\" (UID: \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"
Oct 30 00:54:36.604794 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:36.604744    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e50cc2ce-0447-448d-9c9f-b93131d3fcba-istio-envoy\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn\" (UID: \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"
Oct 30 00:54:36.604956 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:36.604845    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpkwq\" (UniqueName: \"kubernetes.io/projected/e50cc2ce-0447-448d-9c9f-b93131d3fcba-kube-api-access-qpkwq\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn\" (UID: \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"
Oct 30 00:54:36.604956 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:36.604924    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e50cc2ce-0447-448d-9c9f-b93131d3fcba-workload-certs\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn\" (UID: \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"
Oct 30 00:54:36.605067 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:36.604986    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e50cc2ce-0447-448d-9c9f-b93131d3fcba-istiod-ca-cert\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn\" (UID: \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"
Oct 30 00:54:36.605067 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:36.605049    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e50cc2ce-0447-448d-9c9f-b93131d3fcba-credential-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn\" (UID: \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"
Oct 30 00:54:36.605216 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:36.605184    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e50cc2ce-0447-448d-9c9f-b93131d3fcba-istio-data\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn\" (UID: \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"
Oct 30 00:54:36.605282 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:36.605259    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e50cc2ce-0447-448d-9c9f-b93131d3fcba-workload-socket\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn\" (UID: \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"
Oct 30 00:54:36.605584 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:36.605566    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e50cc2ce-0447-448d-9c9f-b93131d3fcba-istiod-ca-cert\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn\" (UID: \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"
Oct 30 00:54:36.607272 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:36.607245    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e50cc2ce-0447-448d-9c9f-b93131d3fcba-istio-envoy\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn\" (UID: \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"
Oct 30 00:54:36.607272 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:36.607261    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e50cc2ce-0447-448d-9c9f-b93131d3fcba-istio-podinfo\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn\" (UID: \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"
Oct 30 00:54:36.627088 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:36.627061    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e50cc2ce-0447-448d-9c9f-b93131d3fcba-istio-token\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn\" (UID: \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"
Oct 30 00:54:36.627315 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:36.627298    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpkwq\" (UniqueName: \"kubernetes.io/projected/e50cc2ce-0447-448d-9c9f-b93131d3fcba-kube-api-access-qpkwq\") pod \"e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn\" (UID: \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\") " pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"
Oct 30 00:54:36.782546 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:36.781074    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"
Oct 30 00:54:36.856814 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:36.856747    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-52785f5e-5b49f86876-8268t"
Oct 30 00:54:37.132514 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:54:37.132482    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode50cc2ce_0447_448d_9c9f_b93131d3fcba.slice/crio-cbb8f58b7a775516d9147d03bdc87714125ea84e5569dd10729c87b25988e048 WatchSource:0}: Error finding container cbb8f58b7a775516d9147d03bdc87714125ea84e5569dd10729c87b25988e048: Status 404 returned error can't find the container with id cbb8f58b7a775516d9147d03bdc87714125ea84e5569dd10729c87b25988e048
Oct 30 00:54:37.133759 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:37.133734    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"]
Oct 30 00:54:37.135077 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:37.135043    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:54:37.135165 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:37.135120    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:54:37.135221 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:37.135163    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:54:37.778556 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:37.778523    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1fa555-7576b49db4-bh5h6"]
Oct 30 00:54:37.830805 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:37.830711    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1fa555-54499dd979-2gck7"]
Oct 30 00:54:37.942691 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:37.942545    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn" event={"ID":"e50cc2ce-0447-448d-9c9f-b93131d3fcba","Type":"ContainerStarted","Data":"743ec3f151e664d2a4f0d686f1da3f1749167f28f42bff20696ed133145f0ad3"}
Oct 30 00:54:37.942691 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:37.942628    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn" event={"ID":"e50cc2ce-0447-448d-9c9f-b93131d3fcba","Type":"ContainerStarted","Data":"cbb8f58b7a775516d9147d03bdc87714125ea84e5569dd10729c87b25988e048"}
Oct 30 00:54:37.991105 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:37.991047    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn" podStartSLOduration=1.9910327140000001 podStartE2EDuration="1.991032714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:37.988609462 +0000 UTC m=+1368.376285852" watchObservedRunningTime="2026-10-30 00:54:37.991032714 +0000 UTC m=+1368.378709103"
Oct 30 00:54:38.556803 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:38.550853    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw"]
Oct 30 00:54:38.573130 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:38.573104    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw"
Oct 30 00:54:38.592349 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:38.592322    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-02fbf5-serving-cert\""
Oct 30 00:54:38.621816 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:38.614835    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw"]
Oct 30 00:54:38.845873 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:38.831947    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxdkv\" (UniqueName: \"kubernetes.io/projected/e207c138-6b79-4a2f-8987-0ecb06fc5503-kube-api-access-jxdkv\") pod \"maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw\" (UID: \"e207c138-6b79-4a2f-8987-0ecb06fc5503\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw"
Oct 30 00:54:38.845873 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:38.832277    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e207c138-6b79-4a2f-8987-0ecb06fc5503-openshift-service-ca\") pod \"maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw\" (UID: \"e207c138-6b79-4a2f-8987-0ecb06fc5503\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw"
Oct 30 00:54:38.845873 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:38.832365    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e207c138-6b79-4a2f-8987-0ecb06fc5503-metrics-certs\") pod \"maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw\" (UID: \"e207c138-6b79-4a2f-8987-0ecb06fc5503\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw"
Oct 30 00:54:38.845873 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:38.832411    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e207c138-6b79-4a2f-8987-0ecb06fc5503-maas-api-tls\") pod \"maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw\" (UID: \"e207c138-6b79-4a2f-8987-0ecb06fc5503\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw"
Oct 30 00:54:38.903317 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:38.901965    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"
Oct 30 00:54:38.957171 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:38.938016    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxdkv\" (UniqueName: \"kubernetes.io/projected/e207c138-6b79-4a2f-8987-0ecb06fc5503-kube-api-access-jxdkv\") pod \"maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw\" (UID: \"e207c138-6b79-4a2f-8987-0ecb06fc5503\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw"
Oct 30 00:54:38.957171 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:38.950770    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e207c138-6b79-4a2f-8987-0ecb06fc5503-openshift-service-ca\") pod \"maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw\" (UID: \"e207c138-6b79-4a2f-8987-0ecb06fc5503\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw"
Oct 30 00:54:38.957171 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:38.950913    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e207c138-6b79-4a2f-8987-0ecb06fc5503-metrics-certs\") pod \"maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw\" (UID: \"e207c138-6b79-4a2f-8987-0ecb06fc5503\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw"
Oct 30 00:54:38.957171 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:38.950974    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e207c138-6b79-4a2f-8987-0ecb06fc5503-maas-api-tls\") pod \"maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw\" (UID: \"e207c138-6b79-4a2f-8987-0ecb06fc5503\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw"
Oct 30 00:54:38.957171 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:54:38.951261    4393 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-02fbf5-serving-cert: secret "maas-api-e2e-api-b-02fbf5-serving-cert" not found
Oct 30 00:54:38.957171 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:54:38.951333    4393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e207c138-6b79-4a2f-8987-0ecb06fc5503-maas-api-tls podName:e207c138-6b79-4a2f-8987-0ecb06fc5503 nodeName:}" failed. No retries permitted until 2026-10-30 00:54:39.451309901 +0000 UTC m=+1369.838986285 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e207c138-6b79-4a2f-8987-0ecb06fc5503-maas-api-tls") pod "maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw" (UID: "e207c138-6b79-4a2f-8987-0ecb06fc5503") : secret "maas-api-e2e-api-b-02fbf5-serving-cert" not found
Oct 30 00:54:38.957171 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:38.952308    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e207c138-6b79-4a2f-8987-0ecb06fc5503-openshift-service-ca\") pod \"maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw\" (UID: \"e207c138-6b79-4a2f-8987-0ecb06fc5503\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw"
Oct 30 00:54:39.002213 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:39.002179    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e207c138-6b79-4a2f-8987-0ecb06fc5503-metrics-certs\") pod \"maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw\" (UID: \"e207c138-6b79-4a2f-8987-0ecb06fc5503\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw"
Oct 30 00:54:39.067852 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:39.067068    4393 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 00:54:39.067852 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:39.067132    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn" podUID="e50cc2ce-0447-448d-9c9f-b93131d3fcba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 00:54:39.121868 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:39.120666    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxdkv\" (UniqueName: \"kubernetes.io/projected/e207c138-6b79-4a2f-8987-0ecb06fc5503-kube-api-access-jxdkv\") pod \"maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw\" (UID: \"e207c138-6b79-4a2f-8987-0ecb06fc5503\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw"
Oct 30 00:54:39.501389 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:39.501294    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e207c138-6b79-4a2f-8987-0ecb06fc5503-maas-api-tls\") pod \"maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw\" (UID: \"e207c138-6b79-4a2f-8987-0ecb06fc5503\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw"
Oct 30 00:54:39.549799 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:39.544351    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e207c138-6b79-4a2f-8987-0ecb06fc5503-maas-api-tls\") pod \"maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw\" (UID: \"e207c138-6b79-4a2f-8987-0ecb06fc5503\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw"
Oct 30 00:54:39.781850 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:39.781754    4393 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 00:54:39.782018 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:39.781843    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn" podUID="e50cc2ce-0447-448d-9c9f-b93131d3fcba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 00:54:39.843415 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:39.843374    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw"
Oct 30 00:54:40.481033 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:40.481005    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw"]
Oct 30 00:54:40.481850 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:54:40.481816    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode207c138_6b79_4a2f_8987_0ecb06fc5503.slice/crio-8fdc4f0edd2a9b3f55d334c309cc466f41a8d76cfdf4895d0e51b6ba0ee2d197 WatchSource:0}: Error finding container 8fdc4f0edd2a9b3f55d334c309cc466f41a8d76cfdf4895d0e51b6ba0ee2d197: Status 404 returned error can't find the container with id 8fdc4f0edd2a9b3f55d334c309cc466f41a8d76cfdf4895d0e51b6ba0ee2d197
Oct 30 00:54:40.790861 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:40.790515    4393 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 00:54:40.790861 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:40.790581    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn" podUID="e50cc2ce-0447-448d-9c9f-b93131d3fcba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 00:54:41.173596 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:41.173563    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw" event={"ID":"e207c138-6b79-4a2f-8987-0ecb06fc5503","Type":"ContainerStarted","Data":"8fdc4f0edd2a9b3f55d334c309cc466f41a8d76cfdf4895d0e51b6ba0ee2d197"}
Oct 30 00:54:41.782344 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:41.782310    4393 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 00:54:41.782743 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:41.782359    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn" podUID="e50cc2ce-0447-448d-9c9f-b93131d3fcba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 00:54:41.852747 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:41.852719    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-6fpr2"
Oct 30 00:54:42.185035 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:42.185000    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw" event={"ID":"e207c138-6b79-4a2f-8987-0ecb06fc5503","Type":"ContainerStarted","Data":"21914ebcf47a671f97437857ac2a3f64d6dca83ba38fcaeb616d7de5af787dc7"}
Oct 30 00:54:42.185246 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:42.185226    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw"
Oct 30 00:54:42.243490 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:42.242548    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw" podStartSLOduration=3.630730524 podStartE2EDuration="4.242524034s" totalImagesPullingTime="611.79351ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:42.23425275 +0000 UTC m=+1372.621929142" watchObservedRunningTime="2026-10-30 00:54:42.242524034 +0000 UTC m=+1372.630200425"
Oct 30 00:54:42.779328 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:42.779297    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-1fa555-7576b49db4-bh5h6" podUID="9d646802-1521-40c0-8f2e-b09016feb486" containerName="payload-processing" containerID="cri-o://19275aeaf09b4d3ddd769039bef5c26c8d2f8c2b1dcee884954b89462874c5fa" gracePeriod=25
Oct 30 00:54:42.782218 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:42.782192    4393 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 00:54:42.782331 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:42.782257    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn" podUID="e50cc2ce-0447-448d-9c9f-b93131d3fcba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 00:54:42.831461 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:42.831426    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fa555-54499dd979-2gck7" podUID="81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3" containerName="payload-pre-processing" containerID="cri-o://b5a00da8aeecbf11d51c540967cb13d521b5a65e428c740d0448fd4907cd9929" gracePeriod=25
Oct 30 00:54:43.195678 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:43.195646    4393 generic.go:356] "Generic (PLEG): container finished" podID="81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3" containerID="b5a00da8aeecbf11d51c540967cb13d521b5a65e428c740d0448fd4907cd9929" exitCode=0
Oct 30 00:54:43.195894 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:43.195738    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fa555-54499dd979-2gck7" event={"ID":"81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3","Type":"ContainerDied","Data":"b5a00da8aeecbf11d51c540967cb13d521b5a65e428c740d0448fd4907cd9929"}
Oct 30 00:54:43.201319 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:43.198051    4393 generic.go:356] "Generic (PLEG): container finished" podID="9d646802-1521-40c0-8f2e-b09016feb486" containerID="19275aeaf09b4d3ddd769039bef5c26c8d2f8c2b1dcee884954b89462874c5fa" exitCode=0
Oct 30 00:54:43.201319 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:43.200532    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1fa555-7576b49db4-bh5h6" event={"ID":"9d646802-1521-40c0-8f2e-b09016feb486","Type":"ContainerDied","Data":"19275aeaf09b4d3ddd769039bef5c26c8d2f8c2b1dcee884954b89462874c5fa"}
Oct 30 00:54:43.337186 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:43.337110    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1fa555-7576b49db4-bh5h6"
Oct 30 00:54:43.342183 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:43.342158    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fa555-54499dd979-2gck7"
Oct 30 00:54:43.371703 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:43.371674    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3-plugins-config-volume\") pod \"81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3\" (UID: \"81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3\") "
Oct 30 00:54:43.371893 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:43.371878    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d646802-1521-40c0-8f2e-b09016feb486-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d646802-1521-40c0-8f2e-b09016feb486-plugins-config-volume\") pod \"9d646802-1521-40c0-8f2e-b09016feb486\" (UID: \"9d646802-1521-40c0-8f2e-b09016feb486\") "
Oct 30 00:54:43.371962 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:43.371914    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d646802-1521-40c0-8f2e-b09016feb486-kube-api-access-p2n8l\" (UniqueName: \"kubernetes.io/projected/9d646802-1521-40c0-8f2e-b09016feb486-kube-api-access-p2n8l\") pod \"9d646802-1521-40c0-8f2e-b09016feb486\" (UID: \"9d646802-1521-40c0-8f2e-b09016feb486\") "
Oct 30 00:54:43.372003 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:43.371960    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3-kube-api-access-6t652\" (UniqueName: \"kubernetes.io/projected/81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3-kube-api-access-6t652\") pod \"81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3\" (UID: \"81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3\") "
Oct 30 00:54:43.372065 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:43.372044    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3-plugins-config-volume" pod "81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3" (UID: "81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:54:43.372357 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:43.372329    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d646802-1521-40c0-8f2e-b09016feb486-plugins-config-volume" pod "9d646802-1521-40c0-8f2e-b09016feb486" (UID: "9d646802-1521-40c0-8f2e-b09016feb486"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:54:43.372891 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:43.372701    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d646802-1521-40c0-8f2e-b09016feb486-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:54:43.372891 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:43.372891    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:54:43.380955 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:43.380201    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d646802-1521-40c0-8f2e-b09016feb486-kube-api-access-p2n8l" pod "9d646802-1521-40c0-8f2e-b09016feb486" (UID: "9d646802-1521-40c0-8f2e-b09016feb486"). InnerVolumeSpecName "kube-api-access-p2n8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:54:43.386645 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:43.386537    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3-kube-api-access-6t652" pod "81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3" (UID: "81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3"). InnerVolumeSpecName "kube-api-access-6t652". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:54:43.474688 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:43.474650    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2n8l\" (UniqueName: \"kubernetes.io/projected/9d646802-1521-40c0-8f2e-b09016feb486-kube-api-access-p2n8l\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:54:43.474688 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:43.474689    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t652\" (UniqueName: \"kubernetes.io/projected/81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3-kube-api-access-6t652\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:54:43.781513 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:43.781483    4393 patch_prober.go:29] interesting pod/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body=
Oct 30 00:54:43.781679 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:43.781553    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn" podUID="e50cc2ce-0447-448d-9c9f-b93131d3fcba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused"
Oct 30 00:54:44.211205 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:44.211175    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fa555-54499dd979-2gck7"
Oct 30 00:54:44.211832 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:44.211194    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fa555-54499dd979-2gck7" event={"ID":"81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3","Type":"ContainerDied","Data":"86e6a32298da0828bbdde1bf3bd0dd0cb17876dba448e04b783be5ba50086b0c"}
Oct 30 00:54:44.211832 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:44.211274    4393 scope.go:122] "RemoveContainer" containerID="b5a00da8aeecbf11d51c540967cb13d521b5a65e428c740d0448fd4907cd9929"
Oct 30 00:54:44.213195 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:44.213172    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1fa555-7576b49db4-bh5h6"
Oct 30 00:54:44.213925 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:44.213894    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1fa555-7576b49db4-bh5h6" event={"ID":"9d646802-1521-40c0-8f2e-b09016feb486","Type":"ContainerDied","Data":"40aacc91c23bb6fbc8a67bbcbbd7837d93a41b710b012a3fe1ebde457822b91d"}
Oct 30 00:54:44.226562 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:44.226538    4393 scope.go:122] "RemoveContainer" containerID="19275aeaf09b4d3ddd769039bef5c26c8d2f8c2b1dcee884954b89462874c5fa"
Oct 30 00:54:44.269421 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:44.269378    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1fa555-54499dd979-2gck7"]
Oct 30 00:54:44.283720 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:44.283689    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1fa555-54499dd979-2gck7"]
Oct 30 00:54:44.305494 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:44.305453    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1fa555-7576b49db4-bh5h6"]
Oct 30 00:54:44.309980 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:44.309953    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1fa555-7576b49db4-bh5h6"]
Oct 30 00:54:44.787531 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:44.787501    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"
Oct 30 00:54:44.788039 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:44.788021    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"
Oct 30 00:54:44.788918 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:44.788898    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"
Oct 30 00:54:45.727441 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:45.727407    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3" path="/var/lib/kubelet/pods/81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3/volumes"
Oct 30 00:54:45.727915 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:45.727899    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d646802-1521-40c0-8f2e-b09016feb486" path="/var/lib/kubelet/pods/9d646802-1521-40c0-8f2e-b09016feb486/volumes"
Oct 30 00:54:48.206579 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:48.206551    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw"
Oct 30 00:54:50.043297 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:50.043263    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt"]
Oct 30 00:54:50.044186 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:50.044162    4393 cpu_manager.go:414] "containerMap: removing container" podUID="81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3" containerName="payload-pre-processing"
Oct 30 00:54:50.044186 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:50.044181    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3" containerName="payload-pre-processing"
Oct 30 00:54:50.044385 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:50.044207    4393 cpu_manager.go:414] "containerMap: removing container" podUID="9d646802-1521-40c0-8f2e-b09016feb486" containerName="payload-processing"
Oct 30 00:54:50.044385 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:50.044212    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d646802-1521-40c0-8f2e-b09016feb486" containerName="payload-processing"
Oct 30 00:54:50.044385 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:50.044320    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d646802-1521-40c0-8f2e-b09016feb486" containerName="payload-processing"
Oct 30 00:54:50.044385 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:50.044337    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="81e86a2b-f8fa-4cf2-b9a5-66f3eacf7fc3" containerName="payload-pre-processing"
Oct 30 00:54:50.050100 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:50.050082    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt"
Oct 30 00:54:50.070124 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:50.070093    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d7a6db25-serving-cert\""
Oct 30 00:54:50.078946 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:50.078917    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt"]
Oct 30 00:54:50.221803 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:50.210866    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8210f9da-c633-443c-a29e-27b23076df42-maas-api-tls\") pod \"maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt\" (UID: \"8210f9da-c633-443c-a29e-27b23076df42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt"
Oct 30 00:54:50.221803 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:50.211013    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvgk7\" (UniqueName: \"kubernetes.io/projected/8210f9da-c633-443c-a29e-27b23076df42-kube-api-access-xvgk7\") pod \"maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt\" (UID: \"8210f9da-c633-443c-a29e-27b23076df42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt"
Oct 30 00:54:50.221803 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:50.211070    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8210f9da-c633-443c-a29e-27b23076df42-metrics-certs\") pod \"maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt\" (UID: \"8210f9da-c633-443c-a29e-27b23076df42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt"
Oct 30 00:54:50.221803 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:50.211137    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8210f9da-c633-443c-a29e-27b23076df42-openshift-service-ca\") pod \"maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt\" (UID: \"8210f9da-c633-443c-a29e-27b23076df42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt"
Oct 30 00:54:50.331919 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:50.329840    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8210f9da-c633-443c-a29e-27b23076df42-maas-api-tls\") pod \"maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt\" (UID: \"8210f9da-c633-443c-a29e-27b23076df42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt"
Oct 30 00:54:50.331919 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:50.331606    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvgk7\" (UniqueName: \"kubernetes.io/projected/8210f9da-c633-443c-a29e-27b23076df42-kube-api-access-xvgk7\") pod \"maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt\" (UID: \"8210f9da-c633-443c-a29e-27b23076df42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt"
Oct 30 00:54:50.331919 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:50.331765    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8210f9da-c633-443c-a29e-27b23076df42-metrics-certs\") pod \"maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt\" (UID: \"8210f9da-c633-443c-a29e-27b23076df42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt"
Oct 30 00:54:50.332182 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:50.332036    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8210f9da-c633-443c-a29e-27b23076df42-openshift-service-ca\") pod \"maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt\" (UID: \"8210f9da-c633-443c-a29e-27b23076df42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt"
Oct 30 00:54:50.335363 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:50.333215    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8210f9da-c633-443c-a29e-27b23076df42-maas-api-tls\") pod \"maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt\" (UID: \"8210f9da-c633-443c-a29e-27b23076df42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt"
Oct 30 00:54:50.335363 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:50.333712    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8210f9da-c633-443c-a29e-27b23076df42-openshift-service-ca\") pod \"maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt\" (UID: \"8210f9da-c633-443c-a29e-27b23076df42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt"
Oct 30 00:54:50.346212 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:50.337200    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8210f9da-c633-443c-a29e-27b23076df42-metrics-certs\") pod \"maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt\" (UID: \"8210f9da-c633-443c-a29e-27b23076df42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt"
Oct 30 00:54:50.407721 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:50.404467    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvgk7\" (UniqueName: \"kubernetes.io/projected/8210f9da-c633-443c-a29e-27b23076df42-kube-api-access-xvgk7\") pod \"maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt\" (UID: \"8210f9da-c633-443c-a29e-27b23076df42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt"
Oct 30 00:54:50.686493 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:50.686459    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt"
Oct 30 00:54:51.030104 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:54:51.030080    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8210f9da_c633_443c_a29e_27b23076df42.slice/crio-8b8b4a89a1aae25c3e035dbd02fc468bfc6e7ec10555bc7ef724364055b20de4 WatchSource:0}: Error finding container 8b8b4a89a1aae25c3e035dbd02fc468bfc6e7ec10555bc7ef724364055b20de4: Status 404 returned error can't find the container with id 8b8b4a89a1aae25c3e035dbd02fc468bfc6e7ec10555bc7ef724364055b20de4
Oct 30 00:54:51.031409 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:51.031387    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt"]
Oct 30 00:54:51.444494 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:51.444451    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt" event={"ID":"8210f9da-c633-443c-a29e-27b23076df42","Type":"ContainerStarted","Data":"8b8b4a89a1aae25c3e035dbd02fc468bfc6e7ec10555bc7ef724364055b20de4"}
Oct 30 00:54:52.454633 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:52.454584    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt" event={"ID":"8210f9da-c633-443c-a29e-27b23076df42","Type":"ContainerStarted","Data":"58130d39bd68964cb4ffa44fc93e681cbac572d0bfaf225ede90fb98d66bf851"}
Oct 30 00:54:52.455070 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:52.454680    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt"
Oct 30 00:54:52.499546 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:52.499489    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt" podStartSLOduration=2.040175971 podStartE2EDuration="2.499472486s" totalImagesPullingTime="459.296515ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:54:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:54:52.494718778 +0000 UTC m=+1382.882395168" watchObservedRunningTime="2026-10-30 00:54:52.499472486 +0000 UTC m=+1382.887148880"
Oct 30 00:54:56.622770 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:54:56.622715    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:54:58.470061 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:54:58.470032    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt"
Oct 30 00:55:16.435391 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:16.435356    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1fa555-d4565947-nh89k"]
Oct 30 00:55:16.436043 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:16.435655    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fa555-d4565947-nh89k" podUID="3ef7dab9-ca94-4f57-85e1-cd0492b23e2b" containerName="maas-api" containerID="cri-o://6dd98110dd39a503696d24243912fd9ae777feb4b9bb0711240c6230494e4a24" gracePeriod=30
Oct 30 00:55:16.877748 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:16.877715    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fa555-d4565947-nh89k"
Oct 30 00:55:17.026857 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:17.025945    4393 generic.go:356] "Generic (PLEG): container finished" podID="3ef7dab9-ca94-4f57-85e1-cd0492b23e2b" containerID="6dd98110dd39a503696d24243912fd9ae777feb4b9bb0711240c6230494e4a24" exitCode=0
Oct 30 00:55:17.026857 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:17.026080    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fa555-d4565947-nh89k" event={"ID":"3ef7dab9-ca94-4f57-85e1-cd0492b23e2b","Type":"ContainerDied","Data":"6dd98110dd39a503696d24243912fd9ae777feb4b9bb0711240c6230494e4a24"}
Oct 30 00:55:17.026857 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:17.026133    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fa555-d4565947-nh89k" event={"ID":"3ef7dab9-ca94-4f57-85e1-cd0492b23e2b","Type":"ContainerDied","Data":"1f987e95a0d402c5be21a091d07f497af6a341a05f13300d20abf1ae7f3699d9"}
Oct 30 00:55:17.026857 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:17.026163    4393 scope.go:122] "RemoveContainer" containerID="6dd98110dd39a503696d24243912fd9ae777feb4b9bb0711240c6230494e4a24"
Oct 30 00:55:17.026857 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:17.026323    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fa555-d4565947-nh89k"
Oct 30 00:55:17.065877 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:17.062006    4393 scope.go:122] "RemoveContainer" containerID="6dd98110dd39a503696d24243912fd9ae777feb4b9bb0711240c6230494e4a24"
Oct 30 00:55:17.065877 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:55:17.062511    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dd98110dd39a503696d24243912fd9ae777feb4b9bb0711240c6230494e4a24\": container with ID starting with 6dd98110dd39a503696d24243912fd9ae777feb4b9bb0711240c6230494e4a24 not found: ID does not exist" containerID="6dd98110dd39a503696d24243912fd9ae777feb4b9bb0711240c6230494e4a24"
Oct 30 00:55:17.065877 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:17.062548    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dd98110dd39a503696d24243912fd9ae777feb4b9bb0711240c6230494e4a24"} err="failed to get container status \"6dd98110dd39a503696d24243912fd9ae777feb4b9bb0711240c6230494e4a24\": rpc error: code = NotFound desc = could not find container \"6dd98110dd39a503696d24243912fd9ae777feb4b9bb0711240c6230494e4a24\": container with ID starting with 6dd98110dd39a503696d24243912fd9ae777feb4b9bb0711240c6230494e4a24 not found: ID does not exist"
Oct 30 00:55:17.090803 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:17.072533    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ef7dab9-ca94-4f57-85e1-cd0492b23e2b-kube-api-access-9xp9k\" (UniqueName: \"kubernetes.io/projected/3ef7dab9-ca94-4f57-85e1-cd0492b23e2b-kube-api-access-9xp9k\") pod \"3ef7dab9-ca94-4f57-85e1-cd0492b23e2b\" (UID: \"3ef7dab9-ca94-4f57-85e1-cd0492b23e2b\") "
Oct 30 00:55:17.090803 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:17.072663    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ef7dab9-ca94-4f57-85e1-cd0492b23e2b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ef7dab9-ca94-4f57-85e1-cd0492b23e2b-maas-api-tls\") pod \"3ef7dab9-ca94-4f57-85e1-cd0492b23e2b\" (UID: \"3ef7dab9-ca94-4f57-85e1-cd0492b23e2b\") "
Oct 30 00:55:17.090803 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:17.072863    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ef7dab9-ca94-4f57-85e1-cd0492b23e2b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ef7dab9-ca94-4f57-85e1-cd0492b23e2b-metrics-certs\") pod \"3ef7dab9-ca94-4f57-85e1-cd0492b23e2b\" (UID: \"3ef7dab9-ca94-4f57-85e1-cd0492b23e2b\") "
Oct 30 00:55:17.090803 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:17.073120    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ef7dab9-ca94-4f57-85e1-cd0492b23e2b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ef7dab9-ca94-4f57-85e1-cd0492b23e2b-openshift-service-ca\") pod \"3ef7dab9-ca94-4f57-85e1-cd0492b23e2b\" (UID: \"3ef7dab9-ca94-4f57-85e1-cd0492b23e2b\") "
Oct 30 00:55:17.128664 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:17.102979    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ef7dab9-ca94-4f57-85e1-cd0492b23e2b-openshift-service-ca" pod "3ef7dab9-ca94-4f57-85e1-cd0492b23e2b" (UID: "3ef7dab9-ca94-4f57-85e1-cd0492b23e2b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:55:17.212805 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:17.207853    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ef7dab9-ca94-4f57-85e1-cd0492b23e2b-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:55:17.282515 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:17.281537    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ef7dab9-ca94-4f57-85e1-cd0492b23e2b-metrics-certs" pod "3ef7dab9-ca94-4f57-85e1-cd0492b23e2b" (UID: "3ef7dab9-ca94-4f57-85e1-cd0492b23e2b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:55:17.282515 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:17.281634    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ef7dab9-ca94-4f57-85e1-cd0492b23e2b-maas-api-tls" pod "3ef7dab9-ca94-4f57-85e1-cd0492b23e2b" (UID: "3ef7dab9-ca94-4f57-85e1-cd0492b23e2b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:55:17.311808 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:17.307985    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef7dab9-ca94-4f57-85e1-cd0492b23e2b-kube-api-access-9xp9k" pod "3ef7dab9-ca94-4f57-85e1-cd0492b23e2b" (UID: "3ef7dab9-ca94-4f57-85e1-cd0492b23e2b"). InnerVolumeSpecName "kube-api-access-9xp9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:55:17.311808 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:17.311152    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ef7dab9-ca94-4f57-85e1-cd0492b23e2b-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:55:17.311808 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:17.311181    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ef7dab9-ca94-4f57-85e1-cd0492b23e2b-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:55:17.311808 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:17.311196    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xp9k\" (UniqueName: \"kubernetes.io/projected/3ef7dab9-ca94-4f57-85e1-cd0492b23e2b-kube-api-access-9xp9k\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:55:17.459806 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:17.458838    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1fa555-d4565947-nh89k"]
Oct 30 00:55:17.472232 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:17.468825    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1fa555-d4565947-nh89k"]
Oct 30 00:55:17.736470 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:17.736432    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ef7dab9-ca94-4f57-85e1-cd0492b23e2b" path="/var/lib/kubelet/pods/3ef7dab9-ca94-4f57-85e1-cd0492b23e2b/volumes"
Oct 30 00:55:26.513202 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:55:26.513157    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:55:36.675805 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:36.675495    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"]
Oct 30 00:55:36.676325 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:36.675814    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5" podUID="39473464-58b6-4589-a88d-eaa60276d7f0" containerName="istio-proxy" containerID="cri-o://453a0f7d160fee1cbe7ccd561a2e8fb46670273e83eb7185355944b4fd71a91e" gracePeriod=30
Oct 30 00:55:37.341921 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:37.341882    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-cngvb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 00:55:37.342255 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:37.342221    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb" podUID="f3bd5875-a930-4985-b820-efe272d30763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 00:55:37.438616 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:37.438570    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"]
Oct 30 00:55:37.439162 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:37.439108    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd" podUID="394e8f0b-d96a-44ec-9cdf-a1266ee49a91" containerName="istio-proxy" containerID="cri-o://2dd986d6f0568810d411879ce7fcbf8688aab0d8704561e359b76b558a298306" gracePeriod=30
Oct 30 00:55:41.904675 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:41.904650    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"
Oct 30 00:55:41.971148 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:41.971078    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39473464-58b6-4589-a88d-eaa60276d7f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39473464-58b6-4589-a88d-eaa60276d7f0-istio-envoy\") pod \"39473464-58b6-4589-a88d-eaa60276d7f0\" (UID: \"39473464-58b6-4589-a88d-eaa60276d7f0\") "
Oct 30 00:55:41.971298 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:41.971218    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39473464-58b6-4589-a88d-eaa60276d7f0-kube-api-access-whrx6\" (UniqueName: \"kubernetes.io/projected/39473464-58b6-4589-a88d-eaa60276d7f0-kube-api-access-whrx6\") pod \"39473464-58b6-4589-a88d-eaa60276d7f0\" (UID: \"39473464-58b6-4589-a88d-eaa60276d7f0\") "
Oct 30 00:55:41.971298 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:41.971273    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39473464-58b6-4589-a88d-eaa60276d7f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39473464-58b6-4589-a88d-eaa60276d7f0-istio-data\") pod \"39473464-58b6-4589-a88d-eaa60276d7f0\" (UID: \"39473464-58b6-4589-a88d-eaa60276d7f0\") "
Oct 30 00:55:41.971395 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:41.971378    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39473464-58b6-4589-a88d-eaa60276d7f0-istio-token\" (UniqueName: \"kubernetes.io/projected/39473464-58b6-4589-a88d-eaa60276d7f0-istio-token\") pod \"39473464-58b6-4589-a88d-eaa60276d7f0\" (UID: \"39473464-58b6-4589-a88d-eaa60276d7f0\") "
Oct 30 00:55:41.971488 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:41.971465    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39473464-58b6-4589-a88d-eaa60276d7f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39473464-58b6-4589-a88d-eaa60276d7f0-credential-socket\") pod \"39473464-58b6-4589-a88d-eaa60276d7f0\" (UID: \"39473464-58b6-4589-a88d-eaa60276d7f0\") "
Oct 30 00:55:41.971668 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:41.971563    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39473464-58b6-4589-a88d-eaa60276d7f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39473464-58b6-4589-a88d-eaa60276d7f0-workload-socket\") pod \"39473464-58b6-4589-a88d-eaa60276d7f0\" (UID: \"39473464-58b6-4589-a88d-eaa60276d7f0\") "
Oct 30 00:55:41.971668 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:41.971629    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39473464-58b6-4589-a88d-eaa60276d7f0-istio-data" pod "39473464-58b6-4589-a88d-eaa60276d7f0" (UID: "39473464-58b6-4589-a88d-eaa60276d7f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:55:41.971828 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:41.971702    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39473464-58b6-4589-a88d-eaa60276d7f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39473464-58b6-4589-a88d-eaa60276d7f0-workload-certs\") pod \"39473464-58b6-4589-a88d-eaa60276d7f0\" (UID: \"39473464-58b6-4589-a88d-eaa60276d7f0\") "
Oct 30 00:55:41.971828 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:41.971768    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39473464-58b6-4589-a88d-eaa60276d7f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39473464-58b6-4589-a88d-eaa60276d7f0-istio-podinfo\") pod \"39473464-58b6-4589-a88d-eaa60276d7f0\" (UID: \"39473464-58b6-4589-a88d-eaa60276d7f0\") "
Oct 30 00:55:41.971942 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:41.971862    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39473464-58b6-4589-a88d-eaa60276d7f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39473464-58b6-4589-a88d-eaa60276d7f0-istiod-ca-cert\") pod \"39473464-58b6-4589-a88d-eaa60276d7f0\" (UID: \"39473464-58b6-4589-a88d-eaa60276d7f0\") "
Oct 30 00:55:41.971942 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:41.971895    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39473464-58b6-4589-a88d-eaa60276d7f0-workload-socket" pod "39473464-58b6-4589-a88d-eaa60276d7f0" (UID: "39473464-58b6-4589-a88d-eaa60276d7f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:55:41.972253 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:41.972224    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39473464-58b6-4589-a88d-eaa60276d7f0-workload-certs" pod "39473464-58b6-4589-a88d-eaa60276d7f0" (UID: "39473464-58b6-4589-a88d-eaa60276d7f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:55:41.972571 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:41.972550    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39473464-58b6-4589-a88d-eaa60276d7f0-istiod-ca-cert" pod "39473464-58b6-4589-a88d-eaa60276d7f0" (UID: "39473464-58b6-4589-a88d-eaa60276d7f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:55:41.973096 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:41.972868    4393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39473464-58b6-4589-a88d-eaa60276d7f0-workload-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:55:41.973096 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:41.972900    4393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39473464-58b6-4589-a88d-eaa60276d7f0-workload-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:55:41.973096 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:41.972914    4393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39473464-58b6-4589-a88d-eaa60276d7f0-istiod-ca-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:55:41.973096 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:41.972928    4393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39473464-58b6-4589-a88d-eaa60276d7f0-istio-data\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:55:41.973335 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:41.973146    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39473464-58b6-4589-a88d-eaa60276d7f0-credential-socket" pod "39473464-58b6-4589-a88d-eaa60276d7f0" (UID: "39473464-58b6-4589-a88d-eaa60276d7f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:55:41.974415 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:41.974389    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39473464-58b6-4589-a88d-eaa60276d7f0-kube-api-access-whrx6" pod "39473464-58b6-4589-a88d-eaa60276d7f0" (UID: "39473464-58b6-4589-a88d-eaa60276d7f0"). InnerVolumeSpecName "kube-api-access-whrx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:55:41.975097 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:41.975071    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39473464-58b6-4589-a88d-eaa60276d7f0-istio-token" pod "39473464-58b6-4589-a88d-eaa60276d7f0" (UID: "39473464-58b6-4589-a88d-eaa60276d7f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:55:41.978880 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:41.978851    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39473464-58b6-4589-a88d-eaa60276d7f0-istio-podinfo" pod "39473464-58b6-4589-a88d-eaa60276d7f0" (UID: "39473464-58b6-4589-a88d-eaa60276d7f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:55:41.980109 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:41.980085    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39473464-58b6-4589-a88d-eaa60276d7f0-istio-envoy" pod "39473464-58b6-4589-a88d-eaa60276d7f0" (UID: "39473464-58b6-4589-a88d-eaa60276d7f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:55:42.081548 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.079689    4393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39473464-58b6-4589-a88d-eaa60276d7f0-istio-podinfo\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:55:42.081548 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.079743    4393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39473464-58b6-4589-a88d-eaa60276d7f0-istio-envoy\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:55:42.081548 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.079758    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whrx6\" (UniqueName: \"kubernetes.io/projected/39473464-58b6-4589-a88d-eaa60276d7f0-kube-api-access-whrx6\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:55:42.081548 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.079770    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39473464-58b6-4589-a88d-eaa60276d7f0-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:55:42.081548 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.080994    4393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39473464-58b6-4589-a88d-eaa60276d7f0-credential-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:55:42.651755 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.651713    4393 generic.go:356] "Generic (PLEG): container finished" podID="39473464-58b6-4589-a88d-eaa60276d7f0" containerID="453a0f7d160fee1cbe7ccd561a2e8fb46670273e83eb7185355944b4fd71a91e" exitCode=0
Oct 30 00:55:42.651953 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.651916    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"
Oct 30 00:55:42.652057 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.652036    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5" event={"ID":"39473464-58b6-4589-a88d-eaa60276d7f0","Type":"ContainerDied","Data":"453a0f7d160fee1cbe7ccd561a2e8fb46670273e83eb7185355944b4fd71a91e"}
Oct 30 00:55:42.652110 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.652090    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5" event={"ID":"39473464-58b6-4589-a88d-eaa60276d7f0","Type":"ContainerDied","Data":"441da9bf7d5f8cd9f9b1eaabcc8c1a24a913014b0015634736345f7640048599"}
Oct 30 00:55:42.652152 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.652119    4393 scope.go:122] "RemoveContainer" containerID="453a0f7d160fee1cbe7ccd561a2e8fb46670273e83eb7185355944b4fd71a91e"
Oct 30 00:55:42.658853 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.658825    4393 generic.go:356] "Generic (PLEG): container finished" podID="394e8f0b-d96a-44ec-9cdf-a1266ee49a91" containerID="2dd986d6f0568810d411879ce7fcbf8688aab0d8704561e359b76b558a298306" exitCode=0
Oct 30 00:55:42.659175 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.659155    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd" event={"ID":"394e8f0b-d96a-44ec-9cdf-a1266ee49a91","Type":"ContainerDied","Data":"2dd986d6f0568810d411879ce7fcbf8688aab0d8704561e359b76b558a298306"}
Oct 30 00:55:42.671869 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.670886    4393 scope.go:122] "RemoveContainer" containerID="453a0f7d160fee1cbe7ccd561a2e8fb46670273e83eb7185355944b4fd71a91e"
Oct 30 00:55:42.671869 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:55:42.671205    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"453a0f7d160fee1cbe7ccd561a2e8fb46670273e83eb7185355944b4fd71a91e\": container with ID starting with 453a0f7d160fee1cbe7ccd561a2e8fb46670273e83eb7185355944b4fd71a91e not found: ID does not exist" containerID="453a0f7d160fee1cbe7ccd561a2e8fb46670273e83eb7185355944b4fd71a91e"
Oct 30 00:55:42.671869 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.671239    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"453a0f7d160fee1cbe7ccd561a2e8fb46670273e83eb7185355944b4fd71a91e"} err="failed to get container status \"453a0f7d160fee1cbe7ccd561a2e8fb46670273e83eb7185355944b4fd71a91e\": rpc error: code = NotFound desc = could not find container \"453a0f7d160fee1cbe7ccd561a2e8fb46670273e83eb7185355944b4fd71a91e\": container with ID starting with 453a0f7d160fee1cbe7ccd561a2e8fb46670273e83eb7185355944b4fd71a91e not found: ID does not exist"
Oct 30 00:55:42.719590 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.719231    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"]
Oct 30 00:55:42.728311 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.728285    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-rttg5"]
Oct 30 00:55:42.759298 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.759275    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"
Oct 30 00:55:42.791079 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.791056    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-workload-certs\") pod \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\" (UID: \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\") "
Oct 30 00:55:42.791179 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.791128    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-istio-podinfo\") pod \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\" (UID: \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\") "
Oct 30 00:55:42.791219 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.791184    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-istiod-ca-cert\") pod \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\" (UID: \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\") "
Oct 30 00:55:42.791260 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.791248    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-kube-api-access-4phr4\" (UniqueName: \"kubernetes.io/projected/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-kube-api-access-4phr4\") pod \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\" (UID: \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\") "
Oct 30 00:55:42.791370 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.791344    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-workload-certs" pod "394e8f0b-d96a-44ec-9cdf-a1266ee49a91" (UID: "394e8f0b-d96a-44ec-9cdf-a1266ee49a91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:55:42.791448 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.791424    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-credential-socket\") pod \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\" (UID: \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\") "
Oct 30 00:55:42.791500 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.791486    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-istio-envoy\") pod \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\" (UID: \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\") "
Oct 30 00:55:42.791556 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.791525    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-workload-socket\") pod \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\" (UID: \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\") "
Oct 30 00:55:42.791612 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.791582    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-istio-token\" (UniqueName: \"kubernetes.io/projected/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-istio-token\") pod \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\" (UID: \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\") "
Oct 30 00:55:42.791668 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.791638    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-istiod-ca-cert" pod "394e8f0b-d96a-44ec-9cdf-a1266ee49a91" (UID: "394e8f0b-d96a-44ec-9cdf-a1266ee49a91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:55:42.791668 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.791647    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-istio-data\") pod \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\" (UID: \"394e8f0b-d96a-44ec-9cdf-a1266ee49a91\") "
Oct 30 00:55:42.791818 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.791691    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-credential-socket" pod "394e8f0b-d96a-44ec-9cdf-a1266ee49a91" (UID: "394e8f0b-d96a-44ec-9cdf-a1266ee49a91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:55:42.791892 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.791852    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-workload-socket" pod "394e8f0b-d96a-44ec-9cdf-a1266ee49a91" (UID: "394e8f0b-d96a-44ec-9cdf-a1266ee49a91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:55:42.792980 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.792794    4393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-workload-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:55:42.792980 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.792822    4393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-istiod-ca-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:55:42.792980 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.792837    4393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-credential-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:55:42.792980 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.792852    4393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-workload-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:55:42.793242 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.793154    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-istio-data" pod "394e8f0b-d96a-44ec-9cdf-a1266ee49a91" (UID: "394e8f0b-d96a-44ec-9cdf-a1266ee49a91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:55:42.794139 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.794114    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-istio-envoy" pod "394e8f0b-d96a-44ec-9cdf-a1266ee49a91" (UID: "394e8f0b-d96a-44ec-9cdf-a1266ee49a91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:55:42.794273 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.794253    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-istio-podinfo" pod "394e8f0b-d96a-44ec-9cdf-a1266ee49a91" (UID: "394e8f0b-d96a-44ec-9cdf-a1266ee49a91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:55:42.794393 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.794324    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-kube-api-access-4phr4" pod "394e8f0b-d96a-44ec-9cdf-a1266ee49a91" (UID: "394e8f0b-d96a-44ec-9cdf-a1266ee49a91"). InnerVolumeSpecName "kube-api-access-4phr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:55:42.794439 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.794395    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-istio-token" pod "394e8f0b-d96a-44ec-9cdf-a1266ee49a91" (UID: "394e8f0b-d96a-44ec-9cdf-a1266ee49a91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:55:42.894175 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.894137    4393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-istio-podinfo\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:55:42.894175 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.894168    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4phr4\" (UniqueName: \"kubernetes.io/projected/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-kube-api-access-4phr4\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:55:42.894175 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.894182    4393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-istio-envoy\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:55:42.894406 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.894194    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:55:42.894406 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:42.894206    4393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/394e8f0b-d96a-44ec-9cdf-a1266ee49a91-istio-data\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:55:43.668181 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:43.668147    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"
Oct 30 00:55:43.668646 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:43.668187    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd" event={"ID":"394e8f0b-d96a-44ec-9cdf-a1266ee49a91","Type":"ContainerDied","Data":"37cc592955e5ec8f31d695af509db5fdec5cf0082828ce3e92e349866987d14e"}
Oct 30 00:55:43.668646 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:43.668250    4393 scope.go:122] "RemoveContainer" containerID="2dd986d6f0568810d411879ce7fcbf8688aab0d8704561e359b76b558a298306"
Oct 30 00:55:43.727471 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:43.727433    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39473464-58b6-4589-a88d-eaa60276d7f0" path="/var/lib/kubelet/pods/39473464-58b6-4589-a88d-eaa60276d7f0/volumes"
Oct 30 00:55:43.735175 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:43.735154    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"]
Oct 30 00:55:43.746151 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:43.746128    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-tlfnd"]
Oct 30 00:55:45.332667 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:45.332626    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-6fpr2"]
Oct 30 00:55:45.728924 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:45.728892    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="394e8f0b-d96a-44ec-9cdf-a1266ee49a91" path="/var/lib/kubelet/pods/394e8f0b-d96a-44ec-9cdf-a1266ee49a91/volumes"
Oct 30 00:55:49.311631 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:49.308964    4393 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-56.ec2.internal" podUID="21539217511a9b702eb77d59ab0b339d" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Oct 30 00:55:49.311631 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:49.309062    4393 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-56.ec2.internal"
Oct 30 00:55:49.920472 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:49.920428    4393 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="haproxy" containerStatusID={"Type":"cri-o","ID":"db9b07011d2f050f681fa96d04710e99dfd894cfae442af0b42780ee1d573d36"} pod="kube-system/kube-apiserver-proxy-ip-10-0-135-56.ec2.internal" containerMessage="Container haproxy failed liveness probe, will be restarted"
Oct 30 00:55:49.920697 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:49.920511    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-56.ec2.internal" podUID="21539217511a9b702eb77d59ab0b339d" containerName="haproxy" containerID="cri-o://db9b07011d2f050f681fa96d04710e99dfd894cfae442af0b42780ee1d573d36" gracePeriod=30
Oct 30 00:55:50.341860 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:50.333861    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-6fpr2" podUID="6f0ffbbc-535f-428b-a078-ddde09364f7e" containerName="payload-processing" containerID="cri-o://8617253c737044aa5cad9c5e4acb24cb90cab2e903616528e33295f04b355d55" gracePeriod=25
Oct 30 00:55:50.755360 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:50.755329    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-6fpr2"
Oct 30 00:55:50.838263 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:50.838232    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f0ffbbc-535f-428b-a078-ddde09364f7e-kube-api-access-wrn8c\" (UniqueName: \"kubernetes.io/projected/6f0ffbbc-535f-428b-a078-ddde09364f7e-kube-api-access-wrn8c\") pod \"6f0ffbbc-535f-428b-a078-ddde09364f7e\" (UID: \"6f0ffbbc-535f-428b-a078-ddde09364f7e\") "
Oct 30 00:55:50.838435 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:50.838271    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f0ffbbc-535f-428b-a078-ddde09364f7e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0ffbbc-535f-428b-a078-ddde09364f7e-plugins-config-volume\") pod \"6f0ffbbc-535f-428b-a078-ddde09364f7e\" (UID: \"6f0ffbbc-535f-428b-a078-ddde09364f7e\") "
Oct 30 00:55:50.838824 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:50.838744    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f0ffbbc-535f-428b-a078-ddde09364f7e-plugins-config-volume" pod "6f0ffbbc-535f-428b-a078-ddde09364f7e" (UID: "6f0ffbbc-535f-428b-a078-ddde09364f7e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:55:50.838974 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:50.838928    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0ffbbc-535f-428b-a078-ddde09364f7e-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:55:50.840760 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:50.840733    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f0ffbbc-535f-428b-a078-ddde09364f7e-kube-api-access-wrn8c" pod "6f0ffbbc-535f-428b-a078-ddde09364f7e" (UID: "6f0ffbbc-535f-428b-a078-ddde09364f7e"). InnerVolumeSpecName "kube-api-access-wrn8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:55:50.929249 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:50.929168    4393 generic.go:356] "Generic (PLEG): container finished" podID="6f0ffbbc-535f-428b-a078-ddde09364f7e" containerID="8617253c737044aa5cad9c5e4acb24cb90cab2e903616528e33295f04b355d55" exitCode=0
Oct 30 00:55:50.929249 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:50.929216    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-6fpr2"
Oct 30 00:55:50.929445 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:50.929219    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-6fpr2" event={"ID":"6f0ffbbc-535f-428b-a078-ddde09364f7e","Type":"ContainerDied","Data":"8617253c737044aa5cad9c5e4acb24cb90cab2e903616528e33295f04b355d55"}
Oct 30 00:55:50.929445 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:50.929337    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-6fpr2" event={"ID":"6f0ffbbc-535f-428b-a078-ddde09364f7e","Type":"ContainerDied","Data":"e4a0c876e46b5f1170deef87f0eef036ff02ee040b915d220b376beca60ac5f4"}
Oct 30 00:55:50.929445 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:50.929365    4393 scope.go:122] "RemoveContainer" containerID="8617253c737044aa5cad9c5e4acb24cb90cab2e903616528e33295f04b355d55"
Oct 30 00:55:50.939945 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:50.939920    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrn8c\" (UniqueName: \"kubernetes.io/projected/6f0ffbbc-535f-428b-a078-ddde09364f7e-kube-api-access-wrn8c\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:55:50.940910 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:50.940897    4393 scope.go:122] "RemoveContainer" containerID="8617253c737044aa5cad9c5e4acb24cb90cab2e903616528e33295f04b355d55"
Oct 30 00:55:50.941171 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:55:50.941153    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8617253c737044aa5cad9c5e4acb24cb90cab2e903616528e33295f04b355d55\": container with ID starting with 8617253c737044aa5cad9c5e4acb24cb90cab2e903616528e33295f04b355d55 not found: ID does not exist" containerID="8617253c737044aa5cad9c5e4acb24cb90cab2e903616528e33295f04b355d55"
Oct 30 00:55:50.941217 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:50.941180    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8617253c737044aa5cad9c5e4acb24cb90cab2e903616528e33295f04b355d55"} err="failed to get container status \"8617253c737044aa5cad9c5e4acb24cb90cab2e903616528e33295f04b355d55\": rpc error: code = NotFound desc = could not find container \"8617253c737044aa5cad9c5e4acb24cb90cab2e903616528e33295f04b355d55\": container with ID starting with 8617253c737044aa5cad9c5e4acb24cb90cab2e903616528e33295f04b355d55 not found: ID does not exist"
Oct 30 00:55:50.981322 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:50.981292    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-6fpr2"]
Oct 30 00:55:50.985961 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:50.985934    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d7a6db25-56b6fb9474-6fpr2"]
Oct 30 00:55:51.727408 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:51.727372    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f0ffbbc-535f-428b-a078-ddde09364f7e" path="/var/lib/kubelet/pods/6f0ffbbc-535f-428b-a078-ddde09364f7e/volumes"
Oct 30 00:55:55.358917 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:55.358880    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt"]
Oct 30 00:55:55.359365 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:55.359151    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt" podUID="8210f9da-c633-443c-a29e-27b23076df42" containerName="maas-api" containerID="cri-o://58130d39bd68964cb4ffa44fc93e681cbac572d0bfaf225ede90fb98d66bf851" gracePeriod=30
Oct 30 00:55:55.681498 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:55.681472    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt"
Oct 30 00:55:55.778814 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:55.770171    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8210f9da-c633-443c-a29e-27b23076df42-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8210f9da-c633-443c-a29e-27b23076df42-openshift-service-ca\") pod \"8210f9da-c633-443c-a29e-27b23076df42\" (UID: \"8210f9da-c633-443c-a29e-27b23076df42\") "
Oct 30 00:55:55.778814 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:55.770269    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8210f9da-c633-443c-a29e-27b23076df42-kube-api-access-xvgk7\" (UniqueName: \"kubernetes.io/projected/8210f9da-c633-443c-a29e-27b23076df42-kube-api-access-xvgk7\") pod \"8210f9da-c633-443c-a29e-27b23076df42\" (UID: \"8210f9da-c633-443c-a29e-27b23076df42\") "
Oct 30 00:55:55.778814 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:55.770323    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8210f9da-c633-443c-a29e-27b23076df42-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8210f9da-c633-443c-a29e-27b23076df42-maas-api-tls\") pod \"8210f9da-c633-443c-a29e-27b23076df42\" (UID: \"8210f9da-c633-443c-a29e-27b23076df42\") "
Oct 30 00:55:55.778814 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:55.770402    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8210f9da-c633-443c-a29e-27b23076df42-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8210f9da-c633-443c-a29e-27b23076df42-metrics-certs\") pod \"8210f9da-c633-443c-a29e-27b23076df42\" (UID: \"8210f9da-c633-443c-a29e-27b23076df42\") "
Oct 30 00:55:55.778814 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:55.774807    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8210f9da-c633-443c-a29e-27b23076df42-openshift-service-ca" pod "8210f9da-c633-443c-a29e-27b23076df42" (UID: "8210f9da-c633-443c-a29e-27b23076df42"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:55:55.882805 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:55.879821    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8210f9da-c633-443c-a29e-27b23076df42-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:55:55.882805 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:55.881529    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8210f9da-c633-443c-a29e-27b23076df42-metrics-certs" pod "8210f9da-c633-443c-a29e-27b23076df42" (UID: "8210f9da-c633-443c-a29e-27b23076df42"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:55:55.882805 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:55.881642    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8210f9da-c633-443c-a29e-27b23076df42-maas-api-tls" pod "8210f9da-c633-443c-a29e-27b23076df42" (UID: "8210f9da-c633-443c-a29e-27b23076df42"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:55:55.903651 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:55.901978    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8210f9da-c633-443c-a29e-27b23076df42-kube-api-access-xvgk7" pod "8210f9da-c633-443c-a29e-27b23076df42" (UID: "8210f9da-c633-443c-a29e-27b23076df42"). InnerVolumeSpecName "kube-api-access-xvgk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:55:55.985850 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:55.983687    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvgk7\" (UniqueName: \"kubernetes.io/projected/8210f9da-c633-443c-a29e-27b23076df42-kube-api-access-xvgk7\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:55:55.985850 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:55.983729    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8210f9da-c633-443c-a29e-27b23076df42-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:55:55.985850 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:55.983747    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8210f9da-c633-443c-a29e-27b23076df42-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:55:56.090094 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:56.090057    4393 generic.go:356] "Generic (PLEG): container finished" podID="8210f9da-c633-443c-a29e-27b23076df42" containerID="58130d39bd68964cb4ffa44fc93e681cbac572d0bfaf225ede90fb98d66bf851" exitCode=0
Oct 30 00:55:56.090401 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:56.090378    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt" event={"ID":"8210f9da-c633-443c-a29e-27b23076df42","Type":"ContainerDied","Data":"58130d39bd68964cb4ffa44fc93e681cbac572d0bfaf225ede90fb98d66bf851"}
Oct 30 00:55:56.090562 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:56.090545    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt" event={"ID":"8210f9da-c633-443c-a29e-27b23076df42","Type":"ContainerDied","Data":"8b8b4a89a1aae25c3e035dbd02fc468bfc6e7ec10555bc7ef724364055b20de4"}
Oct 30 00:55:56.090720 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:56.090687    4393 scope.go:122] "RemoveContainer" containerID="58130d39bd68964cb4ffa44fc93e681cbac572d0bfaf225ede90fb98d66bf851"
Oct 30 00:55:56.091182 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:56.090947    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt"
Oct 30 00:55:56.121325 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:56.121302    4393 scope.go:122] "RemoveContainer" containerID="58130d39bd68964cb4ffa44fc93e681cbac572d0bfaf225ede90fb98d66bf851"
Oct 30 00:55:56.121657 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:55:56.121641    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58130d39bd68964cb4ffa44fc93e681cbac572d0bfaf225ede90fb98d66bf851\": container with ID starting with 58130d39bd68964cb4ffa44fc93e681cbac572d0bfaf225ede90fb98d66bf851 not found: ID does not exist" containerID="58130d39bd68964cb4ffa44fc93e681cbac572d0bfaf225ede90fb98d66bf851"
Oct 30 00:55:56.121711 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:56.121668    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58130d39bd68964cb4ffa44fc93e681cbac572d0bfaf225ede90fb98d66bf851"} err="failed to get container status \"58130d39bd68964cb4ffa44fc93e681cbac572d0bfaf225ede90fb98d66bf851\": rpc error: code = NotFound desc = could not find container \"58130d39bd68964cb4ffa44fc93e681cbac572d0bfaf225ede90fb98d66bf851\": container with ID starting with 58130d39bd68964cb4ffa44fc93e681cbac572d0bfaf225ede90fb98d66bf851 not found: ID does not exist"
Oct 30 00:55:56.175036 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:56.174948    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt"]
Oct 30 00:55:56.196139 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:56.196104    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d7a6db25-56bc6bd7b8-qtzmt"]
Oct 30 00:55:56.563876 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:55:56.563771    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:55:57.907554 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:55:57.907517    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8210f9da-c633-443c-a29e-27b23076df42" path="/var/lib/kubelet/pods/8210f9da-c633-443c-a29e-27b23076df42/volumes"
Oct 30 00:56:03.353663 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:03.353628    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d788ba-7b5d769864-b98xr"]
Oct 30 00:56:03.354192 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:03.353914    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d788ba-7b5d769864-b98xr" podUID="57d02d2c-140f-4463-a8b5-0fdc4dbabcf8" containerName="maas-api" containerID="cri-o://85d9506af8a0ebdcb7f828481960018f8fceaf635d1d81f0ee9dff13d6379238" gracePeriod=30
Oct 30 00:56:04.027863 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:04.019494    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d788ba-7b5d769864-b98xr"
Oct 30 00:56:04.073997 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:04.073959    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57d02d2c-140f-4463-a8b5-0fdc4dbabcf8-kube-api-access-wp5mw\" (UniqueName: \"kubernetes.io/projected/57d02d2c-140f-4463-a8b5-0fdc4dbabcf8-kube-api-access-wp5mw\") pod \"57d02d2c-140f-4463-a8b5-0fdc4dbabcf8\" (UID: \"57d02d2c-140f-4463-a8b5-0fdc4dbabcf8\") "
Oct 30 00:56:04.074965 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:04.074938    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57d02d2c-140f-4463-a8b5-0fdc4dbabcf8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/57d02d2c-140f-4463-a8b5-0fdc4dbabcf8-metrics-certs\") pod \"57d02d2c-140f-4463-a8b5-0fdc4dbabcf8\" (UID: \"57d02d2c-140f-4463-a8b5-0fdc4dbabcf8\") "
Oct 30 00:56:04.076730 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:04.075497    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57d02d2c-140f-4463-a8b5-0fdc4dbabcf8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57d02d2c-140f-4463-a8b5-0fdc4dbabcf8-maas-api-tls\") pod \"57d02d2c-140f-4463-a8b5-0fdc4dbabcf8\" (UID: \"57d02d2c-140f-4463-a8b5-0fdc4dbabcf8\") "
Oct 30 00:56:04.077366 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:04.077347    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57d02d2c-140f-4463-a8b5-0fdc4dbabcf8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57d02d2c-140f-4463-a8b5-0fdc4dbabcf8-openshift-service-ca\") pod \"57d02d2c-140f-4463-a8b5-0fdc4dbabcf8\" (UID: \"57d02d2c-140f-4463-a8b5-0fdc4dbabcf8\") "
Oct 30 00:56:04.078057 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:04.077955    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57d02d2c-140f-4463-a8b5-0fdc4dbabcf8-openshift-service-ca" pod "57d02d2c-140f-4463-a8b5-0fdc4dbabcf8" (UID: "57d02d2c-140f-4463-a8b5-0fdc4dbabcf8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:56:04.079028 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:04.078980    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57d02d2c-140f-4463-a8b5-0fdc4dbabcf8-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:56:04.105537 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:04.105500    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57d02d2c-140f-4463-a8b5-0fdc4dbabcf8-metrics-certs" pod "57d02d2c-140f-4463-a8b5-0fdc4dbabcf8" (UID: "57d02d2c-140f-4463-a8b5-0fdc4dbabcf8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:56:04.106475 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:04.106446    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57d02d2c-140f-4463-a8b5-0fdc4dbabcf8-kube-api-access-wp5mw" pod "57d02d2c-140f-4463-a8b5-0fdc4dbabcf8" (UID: "57d02d2c-140f-4463-a8b5-0fdc4dbabcf8"). InnerVolumeSpecName "kube-api-access-wp5mw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:56:04.110988 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:04.110959    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57d02d2c-140f-4463-a8b5-0fdc4dbabcf8-maas-api-tls" pod "57d02d2c-140f-4463-a8b5-0fdc4dbabcf8" (UID: "57d02d2c-140f-4463-a8b5-0fdc4dbabcf8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:56:04.168762 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:04.168726    4393 generic.go:356] "Generic (PLEG): container finished" podID="57d02d2c-140f-4463-a8b5-0fdc4dbabcf8" containerID="85d9506af8a0ebdcb7f828481960018f8fceaf635d1d81f0ee9dff13d6379238" exitCode=0
Oct 30 00:56:04.168961 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:04.168797    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d788ba-7b5d769864-b98xr" event={"ID":"57d02d2c-140f-4463-a8b5-0fdc4dbabcf8","Type":"ContainerDied","Data":"85d9506af8a0ebdcb7f828481960018f8fceaf635d1d81f0ee9dff13d6379238"}
Oct 30 00:56:04.168961 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:04.168773    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d788ba-7b5d769864-b98xr"
Oct 30 00:56:04.168961 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:04.168854    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d788ba-7b5d769864-b98xr" event={"ID":"57d02d2c-140f-4463-a8b5-0fdc4dbabcf8","Type":"ContainerDied","Data":"c93141e082a3447ccc5e2b5ac65af815a94134d30a9f02fe90b0d8f864b54a9b"}
Oct 30 00:56:04.168961 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:04.168882    4393 scope.go:122] "RemoveContainer" containerID="85d9506af8a0ebdcb7f828481960018f8fceaf635d1d81f0ee9dff13d6379238"
Oct 30 00:56:04.180590 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:04.180568    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57d02d2c-140f-4463-a8b5-0fdc4dbabcf8-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:56:04.180653 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:04.180596    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp5mw\" (UniqueName: \"kubernetes.io/projected/57d02d2c-140f-4463-a8b5-0fdc4dbabcf8-kube-api-access-wp5mw\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:56:04.180653 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:04.180603    4393 scope.go:122] "RemoveContainer" containerID="85d9506af8a0ebdcb7f828481960018f8fceaf635d1d81f0ee9dff13d6379238"
Oct 30 00:56:04.180744 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:04.180608    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57d02d2c-140f-4463-a8b5-0fdc4dbabcf8-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:56:04.180918 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:56:04.180897    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85d9506af8a0ebdcb7f828481960018f8fceaf635d1d81f0ee9dff13d6379238\": container with ID starting with 85d9506af8a0ebdcb7f828481960018f8fceaf635d1d81f0ee9dff13d6379238 not found: ID does not exist" containerID="85d9506af8a0ebdcb7f828481960018f8fceaf635d1d81f0ee9dff13d6379238"
Oct 30 00:56:04.181004 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:04.180923    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85d9506af8a0ebdcb7f828481960018f8fceaf635d1d81f0ee9dff13d6379238"} err="failed to get container status \"85d9506af8a0ebdcb7f828481960018f8fceaf635d1d81f0ee9dff13d6379238\": rpc error: code = NotFound desc = could not find container \"85d9506af8a0ebdcb7f828481960018f8fceaf635d1d81f0ee9dff13d6379238\": container with ID starting with 85d9506af8a0ebdcb7f828481960018f8fceaf635d1d81f0ee9dff13d6379238 not found: ID does not exist"
Oct 30 00:56:04.220109 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:04.220086    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d788ba-7b5d769864-b98xr"]
Oct 30 00:56:04.229760 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:04.229740    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d788ba-7b5d769864-b98xr"]
Oct 30 00:56:05.728575 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:05.728543    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57d02d2c-140f-4463-a8b5-0fdc4dbabcf8" path="/var/lib/kubelet/pods/57d02d2c-140f-4463-a8b5-0fdc4dbabcf8/volumes"
Oct 30 00:56:20.357738 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:20.357650    4393 generic.go:356] "Generic (PLEG): container finished" podID="21539217511a9b702eb77d59ab0b339d" containerID="db9b07011d2f050f681fa96d04710e99dfd894cfae442af0b42780ee1d573d36" exitCode=137
Oct 30 00:56:20.358222 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:20.357727    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-56.ec2.internal" event={"ID":"21539217511a9b702eb77d59ab0b339d","Type":"ContainerDied","Data":"db9b07011d2f050f681fa96d04710e99dfd894cfae442af0b42780ee1d573d36"}
Oct 30 00:56:20.358222 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:20.357821    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-56.ec2.internal" event={"ID":"21539217511a9b702eb77d59ab0b339d","Type":"ContainerStarted","Data":"80632decad21bcbe8163926d77442e402a8e01d92c022ecf7fded183ef38573a"}
Oct 30 00:56:26.537324 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:56:26.537288    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:56:27.922997 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:27.922953    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd"]
Oct 30 00:56:27.923489 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:27.923171    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd" podUID="af96c512-8935-4b9d-86b6-18c374aeb0a7" containerName="maas-api" containerID="cri-o://772db66e0f1eb40eb5ee9b3f16a2128273720bb1eeae132cd0aa10a6a6727c03" gracePeriod=30
Oct 30 00:56:28.238552 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:28.238517    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd"
Oct 30 00:56:28.326595 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:28.326552    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af96c512-8935-4b9d-86b6-18c374aeb0a7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/af96c512-8935-4b9d-86b6-18c374aeb0a7-metrics-certs\") pod \"af96c512-8935-4b9d-86b6-18c374aeb0a7\" (UID: \"af96c512-8935-4b9d-86b6-18c374aeb0a7\") "
Oct 30 00:56:28.326862 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:28.326771    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af96c512-8935-4b9d-86b6-18c374aeb0a7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af96c512-8935-4b9d-86b6-18c374aeb0a7-maas-api-tls\") pod \"af96c512-8935-4b9d-86b6-18c374aeb0a7\" (UID: \"af96c512-8935-4b9d-86b6-18c374aeb0a7\") "
Oct 30 00:56:28.326970 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:28.326931    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af96c512-8935-4b9d-86b6-18c374aeb0a7-kube-api-access-lmx8h\" (UniqueName: \"kubernetes.io/projected/af96c512-8935-4b9d-86b6-18c374aeb0a7-kube-api-access-lmx8h\") pod \"af96c512-8935-4b9d-86b6-18c374aeb0a7\" (UID: \"af96c512-8935-4b9d-86b6-18c374aeb0a7\") "
Oct 30 00:56:28.326970 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:28.326959    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af96c512-8935-4b9d-86b6-18c374aeb0a7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af96c512-8935-4b9d-86b6-18c374aeb0a7-openshift-service-ca\") pod \"af96c512-8935-4b9d-86b6-18c374aeb0a7\" (UID: \"af96c512-8935-4b9d-86b6-18c374aeb0a7\") "
Oct 30 00:56:28.327754 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:28.327729    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af96c512-8935-4b9d-86b6-18c374aeb0a7-openshift-service-ca" pod "af96c512-8935-4b9d-86b6-18c374aeb0a7" (UID: "af96c512-8935-4b9d-86b6-18c374aeb0a7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:56:28.330003 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:28.329973    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af96c512-8935-4b9d-86b6-18c374aeb0a7-metrics-certs" pod "af96c512-8935-4b9d-86b6-18c374aeb0a7" (UID: "af96c512-8935-4b9d-86b6-18c374aeb0a7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:56:28.332987 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:28.332954    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af96c512-8935-4b9d-86b6-18c374aeb0a7-maas-api-tls" pod "af96c512-8935-4b9d-86b6-18c374aeb0a7" (UID: "af96c512-8935-4b9d-86b6-18c374aeb0a7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:56:28.333266 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:28.333236    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af96c512-8935-4b9d-86b6-18c374aeb0a7-kube-api-access-lmx8h" pod "af96c512-8935-4b9d-86b6-18c374aeb0a7" (UID: "af96c512-8935-4b9d-86b6-18c374aeb0a7"). InnerVolumeSpecName "kube-api-access-lmx8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:56:28.432808 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:28.429703    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af96c512-8935-4b9d-86b6-18c374aeb0a7-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:56:28.432808 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:28.429749    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af96c512-8935-4b9d-86b6-18c374aeb0a7-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:56:28.432808 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:28.429765    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmx8h\" (UniqueName: \"kubernetes.io/projected/af96c512-8935-4b9d-86b6-18c374aeb0a7-kube-api-access-lmx8h\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:56:28.432808 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:28.429798    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af96c512-8935-4b9d-86b6-18c374aeb0a7-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:56:28.582277 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:28.581887    4393 generic.go:356] "Generic (PLEG): container finished" podID="af96c512-8935-4b9d-86b6-18c374aeb0a7" containerID="772db66e0f1eb40eb5ee9b3f16a2128273720bb1eeae132cd0aa10a6a6727c03" exitCode=0
Oct 30 00:56:28.582277 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:28.582006    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd" event={"ID":"af96c512-8935-4b9d-86b6-18c374aeb0a7","Type":"ContainerDied","Data":"772db66e0f1eb40eb5ee9b3f16a2128273720bb1eeae132cd0aa10a6a6727c03"}
Oct 30 00:56:28.582277 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:28.582078    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd" event={"ID":"af96c512-8935-4b9d-86b6-18c374aeb0a7","Type":"ContainerDied","Data":"3c2100580c60cde350f93347e8af00c90ab7e21586acaa22332d4ca371dda688"}
Oct 30 00:56:28.582277 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:28.582090    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd"
Oct 30 00:56:28.582277 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:28.582130    4393 scope.go:122] "RemoveContainer" containerID="772db66e0f1eb40eb5ee9b3f16a2128273720bb1eeae132cd0aa10a6a6727c03"
Oct 30 00:56:28.672284 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:28.672257    4393 scope.go:122] "RemoveContainer" containerID="772db66e0f1eb40eb5ee9b3f16a2128273720bb1eeae132cd0aa10a6a6727c03"
Oct 30 00:56:28.672684 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:56:28.672656    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"772db66e0f1eb40eb5ee9b3f16a2128273720bb1eeae132cd0aa10a6a6727c03\": container with ID starting with 772db66e0f1eb40eb5ee9b3f16a2128273720bb1eeae132cd0aa10a6a6727c03 not found: ID does not exist" containerID="772db66e0f1eb40eb5ee9b3f16a2128273720bb1eeae132cd0aa10a6a6727c03"
Oct 30 00:56:28.672730 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:28.672700    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"772db66e0f1eb40eb5ee9b3f16a2128273720bb1eeae132cd0aa10a6a6727c03"} err="failed to get container status \"772db66e0f1eb40eb5ee9b3f16a2128273720bb1eeae132cd0aa10a6a6727c03\": rpc error: code = NotFound desc = could not find container \"772db66e0f1eb40eb5ee9b3f16a2128273720bb1eeae132cd0aa10a6a6727c03\": container with ID starting with 772db66e0f1eb40eb5ee9b3f16a2128273720bb1eeae132cd0aa10a6a6727c03 not found: ID does not exist"
Oct 30 00:56:28.699940 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:28.699904    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd"]
Oct 30 00:56:28.707518 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:28.707489    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bdc256-55fc74956c-mm2vd"]
Oct 30 00:56:29.730991 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:29.730954    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af96c512-8935-4b9d-86b6-18c374aeb0a7" path="/var/lib/kubelet/pods/af96c512-8935-4b9d-86b6-18c374aeb0a7/volumes"
Oct 30 00:56:37.146118 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:37.146039    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"]
Oct 30 00:56:37.146553 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:37.146307    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w" podUID="f26d33aa-399c-46f8-936b-74a7a5a2b392" containerName="istio-proxy" containerID="cri-o://15b95fad515dc7fb206397f0d8097d932d188272302a91faeac9f774fbb0be80" gracePeriod=30
Oct 30 00:56:42.476651 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.476623    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"
Oct 30 00:56:42.598431 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.598379    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f26d33aa-399c-46f8-936b-74a7a5a2b392-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26d33aa-399c-46f8-936b-74a7a5a2b392-istio-podinfo\") pod \"f26d33aa-399c-46f8-936b-74a7a5a2b392\" (UID: \"f26d33aa-399c-46f8-936b-74a7a5a2b392\") "
Oct 30 00:56:42.598704 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.598682    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f26d33aa-399c-46f8-936b-74a7a5a2b392-kube-api-access-k64gv\" (UniqueName: \"kubernetes.io/projected/f26d33aa-399c-46f8-936b-74a7a5a2b392-kube-api-access-k64gv\") pod \"f26d33aa-399c-46f8-936b-74a7a5a2b392\" (UID: \"f26d33aa-399c-46f8-936b-74a7a5a2b392\") "
Oct 30 00:56:42.598904 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.598886    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f26d33aa-399c-46f8-936b-74a7a5a2b392-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26d33aa-399c-46f8-936b-74a7a5a2b392-istiod-ca-cert\") pod \"f26d33aa-399c-46f8-936b-74a7a5a2b392\" (UID: \"f26d33aa-399c-46f8-936b-74a7a5a2b392\") "
Oct 30 00:56:42.599078 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.599062    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26d33aa-399c-46f8-936b-74a7a5a2b392-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26d33aa-399c-46f8-936b-74a7a5a2b392-istio-data\") pod \"f26d33aa-399c-46f8-936b-74a7a5a2b392\" (UID: \"f26d33aa-399c-46f8-936b-74a7a5a2b392\") "
Oct 30 00:56:42.599192 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.599178    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26d33aa-399c-46f8-936b-74a7a5a2b392-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26d33aa-399c-46f8-936b-74a7a5a2b392-istio-envoy\") pod \"f26d33aa-399c-46f8-936b-74a7a5a2b392\" (UID: \"f26d33aa-399c-46f8-936b-74a7a5a2b392\") "
Oct 30 00:56:42.599342 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.599329    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26d33aa-399c-46f8-936b-74a7a5a2b392-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26d33aa-399c-46f8-936b-74a7a5a2b392-workload-socket\") pod \"f26d33aa-399c-46f8-936b-74a7a5a2b392\" (UID: \"f26d33aa-399c-46f8-936b-74a7a5a2b392\") "
Oct 30 00:56:42.599547 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.599533    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26d33aa-399c-46f8-936b-74a7a5a2b392-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26d33aa-399c-46f8-936b-74a7a5a2b392-workload-certs\") pod \"f26d33aa-399c-46f8-936b-74a7a5a2b392\" (UID: \"f26d33aa-399c-46f8-936b-74a7a5a2b392\") "
Oct 30 00:56:42.599667 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.599652    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f26d33aa-399c-46f8-936b-74a7a5a2b392-istio-token\" (UniqueName: \"kubernetes.io/projected/f26d33aa-399c-46f8-936b-74a7a5a2b392-istio-token\") pod \"f26d33aa-399c-46f8-936b-74a7a5a2b392\" (UID: \"f26d33aa-399c-46f8-936b-74a7a5a2b392\") "
Oct 30 00:56:42.599927 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.599893    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26d33aa-399c-46f8-936b-74a7a5a2b392-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26d33aa-399c-46f8-936b-74a7a5a2b392-credential-socket\") pod \"f26d33aa-399c-46f8-936b-74a7a5a2b392\" (UID: \"f26d33aa-399c-46f8-936b-74a7a5a2b392\") "
Oct 30 00:56:42.617187 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.605170    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26d33aa-399c-46f8-936b-74a7a5a2b392-credential-socket" pod "f26d33aa-399c-46f8-936b-74a7a5a2b392" (UID: "f26d33aa-399c-46f8-936b-74a7a5a2b392"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:42.617387 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.609111    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26d33aa-399c-46f8-936b-74a7a5a2b392-workload-certs" pod "f26d33aa-399c-46f8-936b-74a7a5a2b392" (UID: "f26d33aa-399c-46f8-936b-74a7a5a2b392"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:42.617463 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.609413    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f26d33aa-399c-46f8-936b-74a7a5a2b392-istiod-ca-cert" pod "f26d33aa-399c-46f8-936b-74a7a5a2b392" (UID: "f26d33aa-399c-46f8-936b-74a7a5a2b392"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:56:42.617539 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.609474    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26d33aa-399c-46f8-936b-74a7a5a2b392-workload-socket" pod "f26d33aa-399c-46f8-936b-74a7a5a2b392" (UID: "f26d33aa-399c-46f8-936b-74a7a5a2b392"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:42.617539 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.613900    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26d33aa-399c-46f8-936b-74a7a5a2b392-istio-data" pod "f26d33aa-399c-46f8-936b-74a7a5a2b392" (UID: "f26d33aa-399c-46f8-936b-74a7a5a2b392"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:42.618061 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.618034    4393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26d33aa-399c-46f8-936b-74a7a5a2b392-credential-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:56:42.618170 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.618065    4393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26d33aa-399c-46f8-936b-74a7a5a2b392-istiod-ca-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:56:42.618170 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.618078    4393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26d33aa-399c-46f8-936b-74a7a5a2b392-istio-data\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:56:42.618170 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.618090    4393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26d33aa-399c-46f8-936b-74a7a5a2b392-workload-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:56:42.618170 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.618102    4393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26d33aa-399c-46f8-936b-74a7a5a2b392-workload-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:56:42.628954 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.628903    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26d33aa-399c-46f8-936b-74a7a5a2b392-istio-envoy" pod "f26d33aa-399c-46f8-936b-74a7a5a2b392" (UID: "f26d33aa-399c-46f8-936b-74a7a5a2b392"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:42.629798 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.629743    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f26d33aa-399c-46f8-936b-74a7a5a2b392-istio-token" pod "f26d33aa-399c-46f8-936b-74a7a5a2b392" (UID: "f26d33aa-399c-46f8-936b-74a7a5a2b392"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:56:42.630024 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.629990    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f26d33aa-399c-46f8-936b-74a7a5a2b392-istio-podinfo" pod "f26d33aa-399c-46f8-936b-74a7a5a2b392" (UID: "f26d33aa-399c-46f8-936b-74a7a5a2b392"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:56:42.630499 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.630464    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f26d33aa-399c-46f8-936b-74a7a5a2b392-kube-api-access-k64gv" pod "f26d33aa-399c-46f8-936b-74a7a5a2b392" (UID: "f26d33aa-399c-46f8-936b-74a7a5a2b392"). InnerVolumeSpecName "kube-api-access-k64gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:56:42.794859 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.791561    4393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26d33aa-399c-46f8-936b-74a7a5a2b392-istio-podinfo\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:56:42.794859 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.791594    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k64gv\" (UniqueName: \"kubernetes.io/projected/f26d33aa-399c-46f8-936b-74a7a5a2b392-kube-api-access-k64gv\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:56:42.794859 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.791611    4393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26d33aa-399c-46f8-936b-74a7a5a2b392-istio-envoy\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:56:42.794859 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.791624    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f26d33aa-399c-46f8-936b-74a7a5a2b392-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:56:42.887807 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.884945    4393 generic.go:356] "Generic (PLEG): container finished" podID="f26d33aa-399c-46f8-936b-74a7a5a2b392" containerID="15b95fad515dc7fb206397f0d8097d932d188272302a91faeac9f774fbb0be80" exitCode=0
Oct 30 00:56:42.887807 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.885065    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w" event={"ID":"f26d33aa-399c-46f8-936b-74a7a5a2b392","Type":"ContainerDied","Data":"15b95fad515dc7fb206397f0d8097d932d188272302a91faeac9f774fbb0be80"}
Oct 30 00:56:42.887807 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.885115    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w" event={"ID":"f26d33aa-399c-46f8-936b-74a7a5a2b392","Type":"ContainerDied","Data":"f2d3471d7e63ab5f301a66e85a59bca609225d51a73073ebf974d7486e6d4d39"}
Oct 30 00:56:42.887807 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.885144    4393 scope.go:122] "RemoveContainer" containerID="15b95fad515dc7fb206397f0d8097d932d188272302a91faeac9f774fbb0be80"
Oct 30 00:56:42.887807 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.885562    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"
Oct 30 00:56:42.923059 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.923029    4393 scope.go:122] "RemoveContainer" containerID="15b95fad515dc7fb206397f0d8097d932d188272302a91faeac9f774fbb0be80"
Oct 30 00:56:42.923400 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:56:42.923380    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15b95fad515dc7fb206397f0d8097d932d188272302a91faeac9f774fbb0be80\": container with ID starting with 15b95fad515dc7fb206397f0d8097d932d188272302a91faeac9f774fbb0be80 not found: ID does not exist" containerID="15b95fad515dc7fb206397f0d8097d932d188272302a91faeac9f774fbb0be80"
Oct 30 00:56:42.923501 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.923410    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15b95fad515dc7fb206397f0d8097d932d188272302a91faeac9f774fbb0be80"} err="failed to get container status \"15b95fad515dc7fb206397f0d8097d932d188272302a91faeac9f774fbb0be80\": rpc error: code = NotFound desc = could not find container \"15b95fad515dc7fb206397f0d8097d932d188272302a91faeac9f774fbb0be80\": container with ID starting with 15b95fad515dc7fb206397f0d8097d932d188272302a91faeac9f774fbb0be80 not found: ID does not exist"
Oct 30 00:56:42.953029 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.952995    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"]
Oct 30 00:56:42.963855 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:42.963822    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-tbj4w"]
Oct 30 00:56:43.731050 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:43.730677    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f26d33aa-399c-46f8-936b-74a7a5a2b392" path="/var/lib/kubelet/pods/f26d33aa-399c-46f8-936b-74a7a5a2b392/volumes"
Oct 30 00:56:47.868911 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:47.868879    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"]
Oct 30 00:56:47.869391 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:47.869143    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv" podUID="9905328c-20ac-42d2-a90e-e2190360b082" containerName="istio-proxy" containerID="cri-o://a5c434c78e2dfe5d609f4687f3f98626aad6ee32cac4edace135ddb6518b4c09" gracePeriod=30
Oct 30 00:56:53.154758 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:53.154693    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"
Oct 30 00:56:53.206180 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:53.206149    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9905328c-20ac-42d2-a90e-e2190360b082-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9905328c-20ac-42d2-a90e-e2190360b082-workload-socket\") pod \"9905328c-20ac-42d2-a90e-e2190360b082\" (UID: \"9905328c-20ac-42d2-a90e-e2190360b082\") "
Oct 30 00:56:53.206342 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:53.206285    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9905328c-20ac-42d2-a90e-e2190360b082-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9905328c-20ac-42d2-a90e-e2190360b082-istio-envoy\") pod \"9905328c-20ac-42d2-a90e-e2190360b082\" (UID: \"9905328c-20ac-42d2-a90e-e2190360b082\") "
Oct 30 00:56:53.206404 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:53.206369    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9905328c-20ac-42d2-a90e-e2190360b082-istio-token\" (UniqueName: \"kubernetes.io/projected/9905328c-20ac-42d2-a90e-e2190360b082-istio-token\") pod \"9905328c-20ac-42d2-a90e-e2190360b082\" (UID: \"9905328c-20ac-42d2-a90e-e2190360b082\") "
Oct 30 00:56:53.206452 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:53.206428    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9905328c-20ac-42d2-a90e-e2190360b082-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9905328c-20ac-42d2-a90e-e2190360b082-credential-socket\") pod \"9905328c-20ac-42d2-a90e-e2190360b082\" (UID: \"9905328c-20ac-42d2-a90e-e2190360b082\") "
Oct 30 00:56:53.206499 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:53.206446    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9905328c-20ac-42d2-a90e-e2190360b082-workload-socket" pod "9905328c-20ac-42d2-a90e-e2190360b082" (UID: "9905328c-20ac-42d2-a90e-e2190360b082"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:53.206499 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:53.206468    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9905328c-20ac-42d2-a90e-e2190360b082-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9905328c-20ac-42d2-a90e-e2190360b082-istio-data\") pod \"9905328c-20ac-42d2-a90e-e2190360b082\" (UID: \"9905328c-20ac-42d2-a90e-e2190360b082\") "
Oct 30 00:56:53.206578 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:53.206498    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9905328c-20ac-42d2-a90e-e2190360b082-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9905328c-20ac-42d2-a90e-e2190360b082-istio-podinfo\") pod \"9905328c-20ac-42d2-a90e-e2190360b082\" (UID: \"9905328c-20ac-42d2-a90e-e2190360b082\") "
Oct 30 00:56:53.206578 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:53.206526    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9905328c-20ac-42d2-a90e-e2190360b082-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9905328c-20ac-42d2-a90e-e2190360b082-istiod-ca-cert\") pod \"9905328c-20ac-42d2-a90e-e2190360b082\" (UID: \"9905328c-20ac-42d2-a90e-e2190360b082\") "
Oct 30 00:56:53.206663 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:53.206580    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9905328c-20ac-42d2-a90e-e2190360b082-kube-api-access-x459r\" (UniqueName: \"kubernetes.io/projected/9905328c-20ac-42d2-a90e-e2190360b082-kube-api-access-x459r\") pod \"9905328c-20ac-42d2-a90e-e2190360b082\" (UID: \"9905328c-20ac-42d2-a90e-e2190360b082\") "
Oct 30 00:56:53.206663 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:53.206627    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9905328c-20ac-42d2-a90e-e2190360b082-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9905328c-20ac-42d2-a90e-e2190360b082-workload-certs\") pod \"9905328c-20ac-42d2-a90e-e2190360b082\" (UID: \"9905328c-20ac-42d2-a90e-e2190360b082\") "
Oct 30 00:56:53.207123 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:53.207080    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9905328c-20ac-42d2-a90e-e2190360b082-istiod-ca-cert" pod "9905328c-20ac-42d2-a90e-e2190360b082" (UID: "9905328c-20ac-42d2-a90e-e2190360b082"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:56:53.207358 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:53.207326    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9905328c-20ac-42d2-a90e-e2190360b082-workload-certs" pod "9905328c-20ac-42d2-a90e-e2190360b082" (UID: "9905328c-20ac-42d2-a90e-e2190360b082"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:53.207358 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:53.207341    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9905328c-20ac-42d2-a90e-e2190360b082-istio-data" pod "9905328c-20ac-42d2-a90e-e2190360b082" (UID: "9905328c-20ac-42d2-a90e-e2190360b082"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:53.207508 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:53.207467    4393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9905328c-20ac-42d2-a90e-e2190360b082-workload-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:56:53.207508 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:53.207489    4393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9905328c-20ac-42d2-a90e-e2190360b082-workload-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:56:53.207508 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:53.207503    4393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9905328c-20ac-42d2-a90e-e2190360b082-istio-data\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:56:53.207705 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:53.207517    4393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9905328c-20ac-42d2-a90e-e2190360b082-istiod-ca-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:56:53.207705 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:53.207619    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9905328c-20ac-42d2-a90e-e2190360b082-credential-socket" pod "9905328c-20ac-42d2-a90e-e2190360b082" (UID: "9905328c-20ac-42d2-a90e-e2190360b082"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:53.212560 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:53.211351    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9905328c-20ac-42d2-a90e-e2190360b082-istio-envoy" pod "9905328c-20ac-42d2-a90e-e2190360b082" (UID: "9905328c-20ac-42d2-a90e-e2190360b082"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:56:53.223249 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:53.223194    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9905328c-20ac-42d2-a90e-e2190360b082-istio-podinfo" pod "9905328c-20ac-42d2-a90e-e2190360b082" (UID: "9905328c-20ac-42d2-a90e-e2190360b082"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:56:53.235298 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:53.235269    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9905328c-20ac-42d2-a90e-e2190360b082-istio-token" pod "9905328c-20ac-42d2-a90e-e2190360b082" (UID: "9905328c-20ac-42d2-a90e-e2190360b082"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:56:53.237672 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:53.237648    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9905328c-20ac-42d2-a90e-e2190360b082-kube-api-access-x459r" pod "9905328c-20ac-42d2-a90e-e2190360b082" (UID: "9905328c-20ac-42d2-a90e-e2190360b082"). InnerVolumeSpecName "kube-api-access-x459r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:56:53.308570 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:53.308536    4393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9905328c-20ac-42d2-a90e-e2190360b082-istio-envoy\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:56:53.308570 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:53.308567    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9905328c-20ac-42d2-a90e-e2190360b082-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:56:53.308824 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:53.308579    4393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9905328c-20ac-42d2-a90e-e2190360b082-credential-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:56:53.308824 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:53.308593    4393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9905328c-20ac-42d2-a90e-e2190360b082-istio-podinfo\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:56:53.308824 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:53.308604    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x459r\" (UniqueName: \"kubernetes.io/projected/9905328c-20ac-42d2-a90e-e2190360b082-kube-api-access-x459r\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:56:54.019062 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:54.018912    4393 generic.go:356] "Generic (PLEG): container finished" podID="9905328c-20ac-42d2-a90e-e2190360b082" containerID="a5c434c78e2dfe5d609f4687f3f98626aad6ee32cac4edace135ddb6518b4c09" exitCode=0
Oct 30 00:56:54.019299 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:54.019202    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv" event={"ID":"9905328c-20ac-42d2-a90e-e2190360b082","Type":"ContainerDied","Data":"a5c434c78e2dfe5d609f4687f3f98626aad6ee32cac4edace135ddb6518b4c09"}
Oct 30 00:56:54.019299 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:54.019265    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv" event={"ID":"9905328c-20ac-42d2-a90e-e2190360b082","Type":"ContainerDied","Data":"cce95a327873a589dc5e861bd8c5f03b5133ddcbe43dcc760d6b3d73e695cc87"}
Oct 30 00:56:54.019431 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:54.019301    4393 scope.go:122] "RemoveContainer" containerID="a5c434c78e2dfe5d609f4687f3f98626aad6ee32cac4edace135ddb6518b4c09"
Oct 30 00:56:54.019567 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:54.019542    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"
Oct 30 00:56:54.042763 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:54.042327    4393 scope.go:122] "RemoveContainer" containerID="a5c434c78e2dfe5d609f4687f3f98626aad6ee32cac4edace135ddb6518b4c09"
Oct 30 00:56:54.042935 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:56:54.042756    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5c434c78e2dfe5d609f4687f3f98626aad6ee32cac4edace135ddb6518b4c09\": container with ID starting with a5c434c78e2dfe5d609f4687f3f98626aad6ee32cac4edace135ddb6518b4c09 not found: ID does not exist" containerID="a5c434c78e2dfe5d609f4687f3f98626aad6ee32cac4edace135ddb6518b4c09"
Oct 30 00:56:54.042935 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:54.042827    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5c434c78e2dfe5d609f4687f3f98626aad6ee32cac4edace135ddb6518b4c09"} err="failed to get container status \"a5c434c78e2dfe5d609f4687f3f98626aad6ee32cac4edace135ddb6518b4c09\": rpc error: code = NotFound desc = could not find container \"a5c434c78e2dfe5d609f4687f3f98626aad6ee32cac4edace135ddb6518b4c09\": container with ID starting with a5c434c78e2dfe5d609f4687f3f98626aad6ee32cac4edace135ddb6518b4c09 not found: ID does not exist"
Oct 30 00:56:54.086723 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:54.086686    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"]
Oct 30 00:56:54.097710 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:54.097683    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d7a6db25-openshift-default-68896bcdc7-l5shv"]
Oct 30 00:56:55.749006 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:56:55.748970    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9905328c-20ac-42d2-a90e-e2190360b082" path="/var/lib/kubelet/pods/9905328c-20ac-42d2-a90e-e2190360b082/volumes"
Oct 30 00:56:56.512945 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:56:56.512908    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:57:01.451195 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.451164    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g"]
Oct 30 00:57:01.451962 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.451947    4393 cpu_manager.go:414] "containerMap: removing container" podUID="9905328c-20ac-42d2-a90e-e2190360b082" containerName="istio-proxy"
Oct 30 00:57:01.452013 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.451964    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9905328c-20ac-42d2-a90e-e2190360b082" containerName="istio-proxy"
Oct 30 00:57:01.452013 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.451981    4393 cpu_manager.go:414] "containerMap: removing container" podUID="3ef7dab9-ca94-4f57-85e1-cd0492b23e2b" containerName="maas-api"
Oct 30 00:57:01.452013 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.451987    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ef7dab9-ca94-4f57-85e1-cd0492b23e2b" containerName="maas-api"
Oct 30 00:57:01.452013 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.451997    4393 cpu_manager.go:414] "containerMap: removing container" podUID="f26d33aa-399c-46f8-936b-74a7a5a2b392" containerName="istio-proxy"
Oct 30 00:57:01.452013 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.452002    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f26d33aa-399c-46f8-936b-74a7a5a2b392" containerName="istio-proxy"
Oct 30 00:57:01.452013 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.452012    4393 cpu_manager.go:414] "containerMap: removing container" podUID="af96c512-8935-4b9d-86b6-18c374aeb0a7" containerName="maas-api"
Oct 30 00:57:01.452193 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.452016    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af96c512-8935-4b9d-86b6-18c374aeb0a7" containerName="maas-api"
Oct 30 00:57:01.452193 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.452031    4393 cpu_manager.go:414] "containerMap: removing container" podUID="8210f9da-c633-443c-a29e-27b23076df42" containerName="maas-api"
Oct 30 00:57:01.452193 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.452036    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8210f9da-c633-443c-a29e-27b23076df42" containerName="maas-api"
Oct 30 00:57:01.452193 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.452046    4393 cpu_manager.go:414] "containerMap: removing container" podUID="394e8f0b-d96a-44ec-9cdf-a1266ee49a91" containerName="istio-proxy"
Oct 30 00:57:01.452193 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.452050    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="394e8f0b-d96a-44ec-9cdf-a1266ee49a91" containerName="istio-proxy"
Oct 30 00:57:01.452193 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.452060    4393 cpu_manager.go:414] "containerMap: removing container" podUID="57d02d2c-140f-4463-a8b5-0fdc4dbabcf8" containerName="maas-api"
Oct 30 00:57:01.452193 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.452065    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57d02d2c-140f-4463-a8b5-0fdc4dbabcf8" containerName="maas-api"
Oct 30 00:57:01.452193 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.452089    4393 cpu_manager.go:414] "containerMap: removing container" podUID="39473464-58b6-4589-a88d-eaa60276d7f0" containerName="istio-proxy"
Oct 30 00:57:01.452193 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.452093    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39473464-58b6-4589-a88d-eaa60276d7f0" containerName="istio-proxy"
Oct 30 00:57:01.452193 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.452106    4393 cpu_manager.go:414] "containerMap: removing container" podUID="6f0ffbbc-535f-428b-a078-ddde09364f7e" containerName="payload-processing"
Oct 30 00:57:01.452193 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.452111    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f0ffbbc-535f-428b-a078-ddde09364f7e" containerName="payload-processing"
Oct 30 00:57:01.452501 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.452220    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="8210f9da-c633-443c-a29e-27b23076df42" containerName="maas-api"
Oct 30 00:57:01.452501 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.452230    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="394e8f0b-d96a-44ec-9cdf-a1266ee49a91" containerName="istio-proxy"
Oct 30 00:57:01.452501 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.452237    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="57d02d2c-140f-4463-a8b5-0fdc4dbabcf8" containerName="maas-api"
Oct 30 00:57:01.452501 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.452247    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="39473464-58b6-4589-a88d-eaa60276d7f0" containerName="istio-proxy"
Oct 30 00:57:01.452501 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.452254    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f0ffbbc-535f-428b-a078-ddde09364f7e" containerName="payload-processing"
Oct 30 00:57:01.452501 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.452260    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="9905328c-20ac-42d2-a90e-e2190360b082" containerName="istio-proxy"
Oct 30 00:57:01.452501 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.452267    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ef7dab9-ca94-4f57-85e1-cd0492b23e2b" containerName="maas-api"
Oct 30 00:57:01.452501 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.452275    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="f26d33aa-399c-46f8-936b-74a7a5a2b392" containerName="istio-proxy"
Oct 30 00:57:01.452501 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.452280    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="af96c512-8935-4b9d-86b6-18c374aeb0a7" containerName="maas-api"
Oct 30 00:57:01.457745 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.457726    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g"
Oct 30 00:57:01.478905 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.478880    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g"]
Oct 30 00:57:01.489118 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.489090    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-bab19a-serving-cert\""
Oct 30 00:57:01.494634 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.494585    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3385764c-37fe-43ba-afb7-f085f5dfab40-openshift-service-ca\") pod \"maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g\" (UID: \"3385764c-37fe-43ba-afb7-f085f5dfab40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g"
Oct 30 00:57:01.494764 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.494649    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcbfk\" (UniqueName: \"kubernetes.io/projected/3385764c-37fe-43ba-afb7-f085f5dfab40-kube-api-access-dcbfk\") pod \"maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g\" (UID: \"3385764c-37fe-43ba-afb7-f085f5dfab40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g"
Oct 30 00:57:01.494764 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.494734    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3385764c-37fe-43ba-afb7-f085f5dfab40-maas-api-tls\") pod \"maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g\" (UID: \"3385764c-37fe-43ba-afb7-f085f5dfab40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g"
Oct 30 00:57:01.495094 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.495071    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3385764c-37fe-43ba-afb7-f085f5dfab40-metrics-certs\") pod \"maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g\" (UID: \"3385764c-37fe-43ba-afb7-f085f5dfab40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g"
Oct 30 00:57:01.596419 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.596383    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3385764c-37fe-43ba-afb7-f085f5dfab40-metrics-certs\") pod \"maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g\" (UID: \"3385764c-37fe-43ba-afb7-f085f5dfab40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g"
Oct 30 00:57:01.596617 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.596470    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3385764c-37fe-43ba-afb7-f085f5dfab40-openshift-service-ca\") pod \"maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g\" (UID: \"3385764c-37fe-43ba-afb7-f085f5dfab40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g"
Oct 30 00:57:01.596686 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.596638    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcbfk\" (UniqueName: \"kubernetes.io/projected/3385764c-37fe-43ba-afb7-f085f5dfab40-kube-api-access-dcbfk\") pod \"maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g\" (UID: \"3385764c-37fe-43ba-afb7-f085f5dfab40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g"
Oct 30 00:57:01.596839 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.596820    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3385764c-37fe-43ba-afb7-f085f5dfab40-maas-api-tls\") pod \"maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g\" (UID: \"3385764c-37fe-43ba-afb7-f085f5dfab40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g"
Oct 30 00:57:01.597261 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.597233    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3385764c-37fe-43ba-afb7-f085f5dfab40-openshift-service-ca\") pod \"maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g\" (UID: \"3385764c-37fe-43ba-afb7-f085f5dfab40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g"
Oct 30 00:57:01.599217 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.599190    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3385764c-37fe-43ba-afb7-f085f5dfab40-metrics-certs\") pod \"maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g\" (UID: \"3385764c-37fe-43ba-afb7-f085f5dfab40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g"
Oct 30 00:57:01.599610 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.599592    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3385764c-37fe-43ba-afb7-f085f5dfab40-maas-api-tls\") pod \"maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g\" (UID: \"3385764c-37fe-43ba-afb7-f085f5dfab40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g"
Oct 30 00:57:01.619998 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.619965    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcbfk\" (UniqueName: \"kubernetes.io/projected/3385764c-37fe-43ba-afb7-f085f5dfab40-kube-api-access-dcbfk\") pod \"maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g\" (UID: \"3385764c-37fe-43ba-afb7-f085f5dfab40\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g"
Oct 30 00:57:01.786974 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:01.785684    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g"
Oct 30 00:57:02.117017 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:57:02.116950    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3385764c_37fe_43ba_afb7_f085f5dfab40.slice/crio-757d93975193e481956d4a6c6575269354451eae0367d1521c88710f9300af3d WatchSource:0}: Error finding container 757d93975193e481956d4a6c6575269354451eae0367d1521c88710f9300af3d: Status 404 returned error can't find the container with id 757d93975193e481956d4a6c6575269354451eae0367d1521c88710f9300af3d
Oct 30 00:57:02.117463 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:02.117440    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g"]
Oct 30 00:57:02.192087 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:02.192052    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g" event={"ID":"3385764c-37fe-43ba-afb7-f085f5dfab40","Type":"ContainerStarted","Data":"757d93975193e481956d4a6c6575269354451eae0367d1521c88710f9300af3d"}
Oct 30 00:57:03.212655 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:03.212615    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g" event={"ID":"3385764c-37fe-43ba-afb7-f085f5dfab40","Type":"ContainerStarted","Data":"eb158a72650e6f0c36a02a50c655ed185ccd0fcb7498c549e86985ecfabf503e"}
Oct 30 00:57:03.213221 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:03.212723    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g"
Oct 30 00:57:03.265114 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:03.265054    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g" podStartSLOduration=1.949906512 podStartE2EDuration="2.265042063s" totalImagesPullingTime="315.135551ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:03.263597419 +0000 UTC m=+1513.651273808" watchObservedRunningTime="2026-10-30 00:57:03.265042063 +0000 UTC m=+1513.652718453"
Oct 30 00:57:06.219235 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:06.219202    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls"]
Oct 30 00:57:06.233856 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:06.233755    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls"
Oct 30 00:57:06.256142 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:06.256114    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls"]
Oct 30 00:57:06.270872 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:06.270842    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6444d7\"/\"kube-root-ca.crt\""
Oct 30 00:57:06.271016 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:06.270845    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6444d7\"/\"openshift-service-ca.crt\""
Oct 30 00:57:06.282519 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:06.282493    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6444d7\"/\"test-model-6444d7-kserve-self-signed-certs\""
Oct 30 00:57:06.282651 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:06.282559    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6444d7\"/\"default-dockercfg-7468n\""
Oct 30 00:57:06.331162 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:06.331111    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4wjr\" (UniqueName: \"kubernetes.io/projected/6362e34f-f675-4d77-b0b8-08176b9c0f4d-kube-api-access-h4wjr\") pod \"test-model-6444d7-kserve-798d574689-xpmls\" (UID: \"6362e34f-f675-4d77-b0b8-08176b9c0f4d\") " pod="ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls"
Oct 30 00:57:06.331356 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:06.331326    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6362e34f-f675-4d77-b0b8-08176b9c0f4d-home\") pod \"test-model-6444d7-kserve-798d574689-xpmls\" (UID: \"6362e34f-f675-4d77-b0b8-08176b9c0f4d\") " pod="ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls"
Oct 30 00:57:06.331472 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:06.331454    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6362e34f-f675-4d77-b0b8-08176b9c0f4d-model-cache\") pod \"test-model-6444d7-kserve-798d574689-xpmls\" (UID: \"6362e34f-f675-4d77-b0b8-08176b9c0f4d\") " pod="ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls"
Oct 30 00:57:06.331531 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:06.331506    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6362e34f-f675-4d77-b0b8-08176b9c0f4d-dshm\") pod \"test-model-6444d7-kserve-798d574689-xpmls\" (UID: \"6362e34f-f675-4d77-b0b8-08176b9c0f4d\") " pod="ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls"
Oct 30 00:57:06.331742 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:06.331721    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6362e34f-f675-4d77-b0b8-08176b9c0f4d-tls-certs\") pod \"test-model-6444d7-kserve-798d574689-xpmls\" (UID: \"6362e34f-f675-4d77-b0b8-08176b9c0f4d\") " pod="ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls"
Oct 30 00:57:06.331890 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:06.331874    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6362e34f-f675-4d77-b0b8-08176b9c0f4d-tmp-dir\") pod \"test-model-6444d7-kserve-798d574689-xpmls\" (UID: \"6362e34f-f675-4d77-b0b8-08176b9c0f4d\") " pod="ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls"
Oct 30 00:57:06.433810 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:06.433765    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6362e34f-f675-4d77-b0b8-08176b9c0f4d-model-cache\") pod \"test-model-6444d7-kserve-798d574689-xpmls\" (UID: \"6362e34f-f675-4d77-b0b8-08176b9c0f4d\") " pod="ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls"
Oct 30 00:57:06.433987 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:06.433829    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6362e34f-f675-4d77-b0b8-08176b9c0f4d-dshm\") pod \"test-model-6444d7-kserve-798d574689-xpmls\" (UID: \"6362e34f-f675-4d77-b0b8-08176b9c0f4d\") " pod="ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls"
Oct 30 00:57:06.433987 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:06.433949    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6362e34f-f675-4d77-b0b8-08176b9c0f4d-tls-certs\") pod \"test-model-6444d7-kserve-798d574689-xpmls\" (UID: \"6362e34f-f675-4d77-b0b8-08176b9c0f4d\") " pod="ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls"
Oct 30 00:57:06.434097 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:06.434027    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6362e34f-f675-4d77-b0b8-08176b9c0f4d-tmp-dir\") pod \"test-model-6444d7-kserve-798d574689-xpmls\" (UID: \"6362e34f-f675-4d77-b0b8-08176b9c0f4d\") " pod="ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls"
Oct 30 00:57:06.434097 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:06.434092    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4wjr\" (UniqueName: \"kubernetes.io/projected/6362e34f-f675-4d77-b0b8-08176b9c0f4d-kube-api-access-h4wjr\") pod \"test-model-6444d7-kserve-798d574689-xpmls\" (UID: \"6362e34f-f675-4d77-b0b8-08176b9c0f4d\") " pod="ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls"
Oct 30 00:57:06.434211 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:06.434184    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6362e34f-f675-4d77-b0b8-08176b9c0f4d-home\") pod \"test-model-6444d7-kserve-798d574689-xpmls\" (UID: \"6362e34f-f675-4d77-b0b8-08176b9c0f4d\") " pod="ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls"
Oct 30 00:57:06.434293 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:06.434262    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6362e34f-f675-4d77-b0b8-08176b9c0f4d-model-cache\") pod \"test-model-6444d7-kserve-798d574689-xpmls\" (UID: \"6362e34f-f675-4d77-b0b8-08176b9c0f4d\") " pod="ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls"
Oct 30 00:57:06.434385 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:06.434357    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6362e34f-f675-4d77-b0b8-08176b9c0f4d-tmp-dir\") pod \"test-model-6444d7-kserve-798d574689-xpmls\" (UID: \"6362e34f-f675-4d77-b0b8-08176b9c0f4d\") " pod="ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls"
Oct 30 00:57:06.434546 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:06.434518    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6362e34f-f675-4d77-b0b8-08176b9c0f4d-home\") pod \"test-model-6444d7-kserve-798d574689-xpmls\" (UID: \"6362e34f-f675-4d77-b0b8-08176b9c0f4d\") " pod="ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls"
Oct 30 00:57:06.436687 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:06.436557    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6362e34f-f675-4d77-b0b8-08176b9c0f4d-dshm\") pod \"test-model-6444d7-kserve-798d574689-xpmls\" (UID: \"6362e34f-f675-4d77-b0b8-08176b9c0f4d\") " pod="ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls"
Oct 30 00:57:06.437126 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:06.437104    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6362e34f-f675-4d77-b0b8-08176b9c0f4d-tls-certs\") pod \"test-model-6444d7-kserve-798d574689-xpmls\" (UID: \"6362e34f-f675-4d77-b0b8-08176b9c0f4d\") " pod="ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls"
Oct 30 00:57:06.493855 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:06.493456    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4wjr\" (UniqueName: \"kubernetes.io/projected/6362e34f-f675-4d77-b0b8-08176b9c0f4d-kube-api-access-h4wjr\") pod \"test-model-6444d7-kserve-798d574689-xpmls\" (UID: \"6362e34f-f675-4d77-b0b8-08176b9c0f4d\") " pod="ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls"
Oct 30 00:57:06.575831 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:06.567556    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls"
Oct 30 00:57:07.272420 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:07.272320    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls" event={"ID":"6362e34f-f675-4d77-b0b8-08176b9c0f4d","Type":"ContainerStarted","Data":"4a9c41804d30dbbb842f206a381175d06a7e3ad2e8b75f9bf641c1a9fdbdbff7"}
Oct 30 00:57:07.286705 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:07.286668    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls"]
Oct 30 00:57:07.764807 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:07.764028    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-52785f5e-5b49f86876-8268t"]
Oct 30 00:57:07.764807 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:07.764622    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-52785f5e-5b49f86876-8268t" podUID="7f559e0b-04b7-48f8-a365-490524c81a47" containerName="maas-api" containerID="cri-o://4040c0119f4b58dbb75e4125a9fa3c48fe5a39912d88fbee8a037f38caa6a4b2" gracePeriod=30
Oct 30 00:57:08.441598 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:08.441562    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls" event={"ID":"6362e34f-f675-4d77-b0b8-08176b9c0f4d","Type":"ContainerStarted","Data":"30b53c15c1c1a1ee95f0198f39c0c5935b04a8c31b8e9035e0602449ab52d6ff"}
Oct 30 00:57:08.444256 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:08.444230    4393 generic.go:356] "Generic (PLEG): container finished" podID="7f559e0b-04b7-48f8-a365-490524c81a47" containerID="4040c0119f4b58dbb75e4125a9fa3c48fe5a39912d88fbee8a037f38caa6a4b2" exitCode=0
Oct 30 00:57:08.444411 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:08.444394    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-52785f5e-5b49f86876-8268t" event={"ID":"7f559e0b-04b7-48f8-a365-490524c81a47","Type":"ContainerDied","Data":"4040c0119f4b58dbb75e4125a9fa3c48fe5a39912d88fbee8a037f38caa6a4b2"}
Oct 30 00:57:08.498015 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:08.497671    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-52785f5e-5b49f86876-8268t"
Oct 30 00:57:08.498922 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:08.498864    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls" podStartSLOduration=2.49882969 podStartE2EDuration="2.49882969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:08.48925754 +0000 UTC m=+1518.876933929" watchObservedRunningTime="2026-10-30 00:57:08.49882969 +0000 UTC m=+1518.886506080"
Oct 30 00:57:08.522924 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:08.522888    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f559e0b-04b7-48f8-a365-490524c81a47-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f559e0b-04b7-48f8-a365-490524c81a47-maas-api-tls\") pod \"7f559e0b-04b7-48f8-a365-490524c81a47\" (UID: \"7f559e0b-04b7-48f8-a365-490524c81a47\") "
Oct 30 00:57:08.523156 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:08.523000    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f559e0b-04b7-48f8-a365-490524c81a47-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f559e0b-04b7-48f8-a365-490524c81a47-openshift-service-ca\") pod \"7f559e0b-04b7-48f8-a365-490524c81a47\" (UID: \"7f559e0b-04b7-48f8-a365-490524c81a47\") "
Oct 30 00:57:08.523237 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:08.523174    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f559e0b-04b7-48f8-a365-490524c81a47-kube-api-access-s8p9k\" (UniqueName: \"kubernetes.io/projected/7f559e0b-04b7-48f8-a365-490524c81a47-kube-api-access-s8p9k\") pod \"7f559e0b-04b7-48f8-a365-490524c81a47\" (UID: \"7f559e0b-04b7-48f8-a365-490524c81a47\") "
Oct 30 00:57:08.523294 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:08.523249    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f559e0b-04b7-48f8-a365-490524c81a47-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f559e0b-04b7-48f8-a365-490524c81a47-metrics-certs\") pod \"7f559e0b-04b7-48f8-a365-490524c81a47\" (UID: \"7f559e0b-04b7-48f8-a365-490524c81a47\") "
Oct 30 00:57:08.523407 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:08.523375    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f559e0b-04b7-48f8-a365-490524c81a47-openshift-service-ca" pod "7f559e0b-04b7-48f8-a365-490524c81a47" (UID: "7f559e0b-04b7-48f8-a365-490524c81a47"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:57:08.524061 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:08.524037    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f559e0b-04b7-48f8-a365-490524c81a47-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:57:08.526269 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:08.526244    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f559e0b-04b7-48f8-a365-490524c81a47-metrics-certs" pod "7f559e0b-04b7-48f8-a365-490524c81a47" (UID: "7f559e0b-04b7-48f8-a365-490524c81a47"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:57:08.526381 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:08.526269    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f559e0b-04b7-48f8-a365-490524c81a47-maas-api-tls" pod "7f559e0b-04b7-48f8-a365-490524c81a47" (UID: "7f559e0b-04b7-48f8-a365-490524c81a47"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:57:08.526381 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:08.526328    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f559e0b-04b7-48f8-a365-490524c81a47-kube-api-access-s8p9k" pod "7f559e0b-04b7-48f8-a365-490524c81a47" (UID: "7f559e0b-04b7-48f8-a365-490524c81a47"). InnerVolumeSpecName "kube-api-access-s8p9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:57:08.568516 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:08.568484    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls"
Oct 30 00:57:08.625630 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:08.625600    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f559e0b-04b7-48f8-a365-490524c81a47-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:57:08.625630 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:08.625625    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8p9k\" (UniqueName: \"kubernetes.io/projected/7f559e0b-04b7-48f8-a365-490524c81a47-kube-api-access-s8p9k\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:57:08.625630 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:08.625634    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f559e0b-04b7-48f8-a365-490524c81a47-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:57:09.226005 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:09.225975    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g"
Oct 30 00:57:09.455460 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:09.455432    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-52785f5e-5b49f86876-8268t"
Oct 30 00:57:09.455924 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:09.455469    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-52785f5e-5b49f86876-8268t" event={"ID":"7f559e0b-04b7-48f8-a365-490524c81a47","Type":"ContainerDied","Data":"326334d9fda56f705bc39e2221807ea55a79d242ff6027f2645d265c5e4f7816"}
Oct 30 00:57:09.455924 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:09.455531    4393 scope.go:122] "RemoveContainer" containerID="4040c0119f4b58dbb75e4125a9fa3c48fe5a39912d88fbee8a037f38caa6a4b2"
Oct 30 00:57:09.519009 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:09.518932    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-52785f5e-5b49f86876-8268t"]
Oct 30 00:57:09.526533 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:09.526498    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-52785f5e-5b49f86876-8268t"]
Oct 30 00:57:09.727005 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:09.726977    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f559e0b-04b7-48f8-a365-490524c81a47" path="/var/lib/kubelet/pods/7f559e0b-04b7-48f8-a365-490524c81a47/volumes"
Oct 30 00:57:12.676592 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:12.676555    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf"]
Oct 30 00:57:12.677450 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:12.677432    4393 cpu_manager.go:414] "containerMap: removing container" podUID="7f559e0b-04b7-48f8-a365-490524c81a47" containerName="maas-api"
Oct 30 00:57:12.677450 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:12.677451    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f559e0b-04b7-48f8-a365-490524c81a47" containerName="maas-api"
Oct 30 00:57:12.677585 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:12.677576    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f559e0b-04b7-48f8-a365-490524c81a47" containerName="maas-api"
Oct 30 00:57:12.682614 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:12.682594    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf"
Oct 30 00:57:12.700847 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:12.700823    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf"]
Oct 30 00:57:12.702680 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:12.702651    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7357139-9713-42ac-bdbe-c9093760765b-metrics-certs\") pod \"maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf\" (UID: \"b7357139-9713-42ac-bdbe-c9093760765b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf"
Oct 30 00:57:12.702970 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:12.702936    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv9g2\" (UniqueName: \"kubernetes.io/projected/b7357139-9713-42ac-bdbe-c9093760765b-kube-api-access-sv9g2\") pod \"maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf\" (UID: \"b7357139-9713-42ac-bdbe-c9093760765b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf"
Oct 30 00:57:12.703068 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:12.703028    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7357139-9713-42ac-bdbe-c9093760765b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf\" (UID: \"b7357139-9713-42ac-bdbe-c9093760765b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf"
Oct 30 00:57:12.703228 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:12.703209    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7357139-9713-42ac-bdbe-c9093760765b-maas-api-tls\") pod \"maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf\" (UID: \"b7357139-9713-42ac-bdbe-c9093760765b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf"
Oct 30 00:57:12.707859 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:12.707833    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-481ab7-serving-cert\""
Oct 30 00:57:12.805151 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:12.805114    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv9g2\" (UniqueName: \"kubernetes.io/projected/b7357139-9713-42ac-bdbe-c9093760765b-kube-api-access-sv9g2\") pod \"maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf\" (UID: \"b7357139-9713-42ac-bdbe-c9093760765b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf"
Oct 30 00:57:12.805411 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:12.805382    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7357139-9713-42ac-bdbe-c9093760765b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf\" (UID: \"b7357139-9713-42ac-bdbe-c9093760765b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf"
Oct 30 00:57:12.805852 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:12.805817    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7357139-9713-42ac-bdbe-c9093760765b-maas-api-tls\") pod \"maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf\" (UID: \"b7357139-9713-42ac-bdbe-c9093760765b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf"
Oct 30 00:57:12.805982 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:57:12.805917    4393 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-481ab7-serving-cert: secret "maas-api-e2e-inf-a-481ab7-serving-cert" not found
Oct 30 00:57:12.806260 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:57:12.805972    4393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7357139-9713-42ac-bdbe-c9093760765b-maas-api-tls podName:b7357139-9713-42ac-bdbe-c9093760765b nodeName:}" failed. No retries permitted until 2026-10-30 00:57:13.305952403 +0000 UTC m=+1523.693628784 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b7357139-9713-42ac-bdbe-c9093760765b-maas-api-tls") pod "maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf" (UID: "b7357139-9713-42ac-bdbe-c9093760765b") : secret "maas-api-e2e-inf-a-481ab7-serving-cert" not found
Oct 30 00:57:12.806355 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:12.806257    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7357139-9713-42ac-bdbe-c9093760765b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf\" (UID: \"b7357139-9713-42ac-bdbe-c9093760765b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf"
Oct 30 00:57:12.806888 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:12.806862    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7357139-9713-42ac-bdbe-c9093760765b-metrics-certs\") pod \"maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf\" (UID: \"b7357139-9713-42ac-bdbe-c9093760765b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf"
Oct 30 00:57:12.809545 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:12.809519    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7357139-9713-42ac-bdbe-c9093760765b-metrics-certs\") pod \"maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf\" (UID: \"b7357139-9713-42ac-bdbe-c9093760765b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf"
Oct 30 00:57:12.827674 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:12.827652    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv9g2\" (UniqueName: \"kubernetes.io/projected/b7357139-9713-42ac-bdbe-c9093760765b-kube-api-access-sv9g2\") pod \"maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf\" (UID: \"b7357139-9713-42ac-bdbe-c9093760765b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf"
Oct 30 00:57:13.316575 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:13.316537    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7357139-9713-42ac-bdbe-c9093760765b-maas-api-tls\") pod \"maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf\" (UID: \"b7357139-9713-42ac-bdbe-c9093760765b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf"
Oct 30 00:57:13.319315 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:13.319288    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7357139-9713-42ac-bdbe-c9093760765b-maas-api-tls\") pod \"maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf\" (UID: \"b7357139-9713-42ac-bdbe-c9093760765b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf"
Oct 30 00:57:13.597591 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:13.597504    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf"
Oct 30 00:57:13.914244 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:13.914214    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf"]
Oct 30 00:57:13.914244 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:57:13.914229    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7357139_9713_42ac_bdbe_c9093760765b.slice/crio-ce90caf9ee63bf61889d4dacfd191be87ec33cca103f4fea92bc8fcfe7dafa61 WatchSource:0}: Error finding container ce90caf9ee63bf61889d4dacfd191be87ec33cca103f4fea92bc8fcfe7dafa61: Status 404 returned error can't find the container with id ce90caf9ee63bf61889d4dacfd191be87ec33cca103f4fea92bc8fcfe7dafa61
Oct 30 00:57:14.498892 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:14.498867    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf" event={"ID":"b7357139-9713-42ac-bdbe-c9093760765b","Type":"ContainerStarted","Data":"ce90caf9ee63bf61889d4dacfd191be87ec33cca103f4fea92bc8fcfe7dafa61"}
Oct 30 00:57:15.508561 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:15.508519    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf" event={"ID":"b7357139-9713-42ac-bdbe-c9093760765b","Type":"ContainerStarted","Data":"03fc326520385a5bfe13f7785cdda51c141845a8c12d76ad348c1c6115edf429"}
Oct 30 00:57:15.508999 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:15.508611    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf"
Oct 30 00:57:15.551066 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:15.551019    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf" podStartSLOduration=3.048432619 podStartE2EDuration="3.55100464s" totalImagesPullingTime="502.572021ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:15.550473263 +0000 UTC m=+1525.938149654" watchObservedRunningTime="2026-10-30 00:57:15.55100464 +0000 UTC m=+1525.938681028"
Oct 30 00:57:16.568714 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:16.568676    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls"
Oct 30 00:57:16.581739 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:16.581711    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls"
Oct 30 00:57:16.941415 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:16.941383    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2"]
Oct 30 00:57:16.946836 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:16.946812    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2"
Oct 30 00:57:16.970447 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:16.969516    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-8dbe73-serving-cert\""
Oct 30 00:57:16.977995 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:16.977966    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2"]
Oct 30 00:57:17.024451 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:17.024415    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016c0a87-3f64-4eb2-ad53-925be83cdf21-maas-api-tls\") pod \"maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2\" (UID: \"016c0a87-3f64-4eb2-ad53-925be83cdf21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2"
Oct 30 00:57:17.024613 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:17.024480    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/016c0a87-3f64-4eb2-ad53-925be83cdf21-metrics-certs\") pod \"maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2\" (UID: \"016c0a87-3f64-4eb2-ad53-925be83cdf21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2"
Oct 30 00:57:17.024613 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:17.024583    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/016c0a87-3f64-4eb2-ad53-925be83cdf21-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2\" (UID: \"016c0a87-3f64-4eb2-ad53-925be83cdf21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2"
Oct 30 00:57:17.024817 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:17.024764    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw7vd\" (UniqueName: \"kubernetes.io/projected/016c0a87-3f64-4eb2-ad53-925be83cdf21-kube-api-access-jw7vd\") pod \"maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2\" (UID: \"016c0a87-3f64-4eb2-ad53-925be83cdf21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2"
Oct 30 00:57:17.126948 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:17.126904    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw7vd\" (UniqueName: \"kubernetes.io/projected/016c0a87-3f64-4eb2-ad53-925be83cdf21-kube-api-access-jw7vd\") pod \"maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2\" (UID: \"016c0a87-3f64-4eb2-ad53-925be83cdf21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2"
Oct 30 00:57:17.127137 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:17.127116    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016c0a87-3f64-4eb2-ad53-925be83cdf21-maas-api-tls\") pod \"maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2\" (UID: \"016c0a87-3f64-4eb2-ad53-925be83cdf21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2"
Oct 30 00:57:17.127189 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:17.127179    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/016c0a87-3f64-4eb2-ad53-925be83cdf21-metrics-certs\") pod \"maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2\" (UID: \"016c0a87-3f64-4eb2-ad53-925be83cdf21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2"
Oct 30 00:57:17.127280 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:57:17.127255    4393 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-8dbe73-serving-cert: secret "maas-api-e2e-inf-a-8dbe73-serving-cert" not found
Oct 30 00:57:17.127421 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:17.127286    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/016c0a87-3f64-4eb2-ad53-925be83cdf21-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2\" (UID: \"016c0a87-3f64-4eb2-ad53-925be83cdf21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2"
Oct 30 00:57:17.127421 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:57:17.127328    4393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/016c0a87-3f64-4eb2-ad53-925be83cdf21-maas-api-tls podName:016c0a87-3f64-4eb2-ad53-925be83cdf21 nodeName:}" failed. No retries permitted until 2026-10-30 00:57:17.627306353 +0000 UTC m=+1528.014982740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/016c0a87-3f64-4eb2-ad53-925be83cdf21-maas-api-tls") pod "maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2" (UID: "016c0a87-3f64-4eb2-ad53-925be83cdf21") : secret "maas-api-e2e-inf-a-8dbe73-serving-cert" not found
Oct 30 00:57:17.128174 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:17.128149    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/016c0a87-3f64-4eb2-ad53-925be83cdf21-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2\" (UID: \"016c0a87-3f64-4eb2-ad53-925be83cdf21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2"
Oct 30 00:57:17.129964 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:17.129940    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/016c0a87-3f64-4eb2-ad53-925be83cdf21-metrics-certs\") pod \"maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2\" (UID: \"016c0a87-3f64-4eb2-ad53-925be83cdf21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2"
Oct 30 00:57:17.207861 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:17.203556    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw7vd\" (UniqueName: \"kubernetes.io/projected/016c0a87-3f64-4eb2-ad53-925be83cdf21-kube-api-access-jw7vd\") pod \"maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2\" (UID: \"016c0a87-3f64-4eb2-ad53-925be83cdf21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2"
Oct 30 00:57:17.545913 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:17.545828    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls"
Oct 30 00:57:17.641105 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:17.641070    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016c0a87-3f64-4eb2-ad53-925be83cdf21-maas-api-tls\") pod \"maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2\" (UID: \"016c0a87-3f64-4eb2-ad53-925be83cdf21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2"
Oct 30 00:57:17.643636 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:17.643616    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016c0a87-3f64-4eb2-ad53-925be83cdf21-maas-api-tls\") pod \"maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2\" (UID: \"016c0a87-3f64-4eb2-ad53-925be83cdf21\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2"
Oct 30 00:57:17.882805 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:17.882267    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2"
Oct 30 00:57:18.236484 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:57:18.236445    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod016c0a87_3f64_4eb2_ad53_925be83cdf21.slice/crio-e972bea36f85864f0025929d86c9396271b448512db6f604534becfc4ab74c5d WatchSource:0}: Error finding container e972bea36f85864f0025929d86c9396271b448512db6f604534becfc4ab74c5d: Status 404 returned error can't find the container with id e972bea36f85864f0025929d86c9396271b448512db6f604534becfc4ab74c5d
Oct 30 00:57:18.237000 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:18.236975    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2"]
Oct 30 00:57:18.553288 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:18.553153    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2" event={"ID":"016c0a87-3f64-4eb2-ad53-925be83cdf21","Type":"ContainerStarted","Data":"e972bea36f85864f0025929d86c9396271b448512db6f604534becfc4ab74c5d"}
Oct 30 00:57:19.561491 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:19.561452    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2" event={"ID":"016c0a87-3f64-4eb2-ad53-925be83cdf21","Type":"ContainerStarted","Data":"88e56eec75a179054b6f9cbe6461eeab23eaf2174766c8f58ac678570f367365"}
Oct 30 00:57:19.562045 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:19.561662    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2"
Oct 30 00:57:19.604818 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:19.604726    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2" podStartSLOduration=3.185128323 podStartE2EDuration="3.604712468s" totalImagesPullingTime="419.584145ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:19.602279618 +0000 UTC m=+1529.989956007" watchObservedRunningTime="2026-10-30 00:57:19.604712468 +0000 UTC m=+1529.992388902"
Oct 30 00:57:20.704916 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:20.704887    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw"]
Oct 30 00:57:20.705370 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:20.705091    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw" podUID="e207c138-6b79-4a2f-8987-0ecb06fc5503" containerName="maas-api" containerID="cri-o://21914ebcf47a671f97437857ac2a3f64d6dca83ba38fcaeb616d7de5af787dc7" gracePeriod=30
Oct 30 00:57:21.473568 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:21.473536    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw"
Oct 30 00:57:21.523844 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:21.523192    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf"
Oct 30 00:57:21.537520 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:21.537486    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e207c138-6b79-4a2f-8987-0ecb06fc5503-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e207c138-6b79-4a2f-8987-0ecb06fc5503-openshift-service-ca\") pod \"e207c138-6b79-4a2f-8987-0ecb06fc5503\" (UID: \"e207c138-6b79-4a2f-8987-0ecb06fc5503\") "
Oct 30 00:57:21.537685 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:21.537593    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e207c138-6b79-4a2f-8987-0ecb06fc5503-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e207c138-6b79-4a2f-8987-0ecb06fc5503-metrics-certs\") pod \"e207c138-6b79-4a2f-8987-0ecb06fc5503\" (UID: \"e207c138-6b79-4a2f-8987-0ecb06fc5503\") "
Oct 30 00:57:21.537685 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:21.537646    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e207c138-6b79-4a2f-8987-0ecb06fc5503-kube-api-access-jxdkv\" (UniqueName: \"kubernetes.io/projected/e207c138-6b79-4a2f-8987-0ecb06fc5503-kube-api-access-jxdkv\") pod \"e207c138-6b79-4a2f-8987-0ecb06fc5503\" (UID: \"e207c138-6b79-4a2f-8987-0ecb06fc5503\") "
Oct 30 00:57:21.537888 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:21.537719    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e207c138-6b79-4a2f-8987-0ecb06fc5503-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e207c138-6b79-4a2f-8987-0ecb06fc5503-maas-api-tls\") pod \"e207c138-6b79-4a2f-8987-0ecb06fc5503\" (UID: \"e207c138-6b79-4a2f-8987-0ecb06fc5503\") "
Oct 30 00:57:21.537968 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:21.537879    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e207c138-6b79-4a2f-8987-0ecb06fc5503-openshift-service-ca" pod "e207c138-6b79-4a2f-8987-0ecb06fc5503" (UID: "e207c138-6b79-4a2f-8987-0ecb06fc5503"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:57:21.538653 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:21.538631    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e207c138-6b79-4a2f-8987-0ecb06fc5503-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:57:21.540681 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:21.540656    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e207c138-6b79-4a2f-8987-0ecb06fc5503-kube-api-access-jxdkv" pod "e207c138-6b79-4a2f-8987-0ecb06fc5503" (UID: "e207c138-6b79-4a2f-8987-0ecb06fc5503"). InnerVolumeSpecName "kube-api-access-jxdkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:57:21.540837 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:21.540677    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e207c138-6b79-4a2f-8987-0ecb06fc5503-maas-api-tls" pod "e207c138-6b79-4a2f-8987-0ecb06fc5503" (UID: "e207c138-6b79-4a2f-8987-0ecb06fc5503"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:57:21.540837 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:21.540693    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e207c138-6b79-4a2f-8987-0ecb06fc5503-metrics-certs" pod "e207c138-6b79-4a2f-8987-0ecb06fc5503" (UID: "e207c138-6b79-4a2f-8987-0ecb06fc5503"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 00:57:21.582761 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:21.582726    4393 generic.go:356] "Generic (PLEG): container finished" podID="e207c138-6b79-4a2f-8987-0ecb06fc5503" containerID="21914ebcf47a671f97437857ac2a3f64d6dca83ba38fcaeb616d7de5af787dc7" exitCode=0
Oct 30 00:57:21.582955 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:21.582825    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw"
Oct 30 00:57:21.582955 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:21.582856    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw" event={"ID":"e207c138-6b79-4a2f-8987-0ecb06fc5503","Type":"ContainerDied","Data":"21914ebcf47a671f97437857ac2a3f64d6dca83ba38fcaeb616d7de5af787dc7"}
Oct 30 00:57:21.583384 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:21.583358    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw" event={"ID":"e207c138-6b79-4a2f-8987-0ecb06fc5503","Type":"ContainerDied","Data":"8fdc4f0edd2a9b3f55d334c309cc466f41a8d76cfdf4895d0e51b6ba0ee2d197"}
Oct 30 00:57:21.583484 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:21.583407    4393 scope.go:122] "RemoveContainer" containerID="21914ebcf47a671f97437857ac2a3f64d6dca83ba38fcaeb616d7de5af787dc7"
Oct 30 00:57:21.600517 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:21.600490    4393 scope.go:122] "RemoveContainer" containerID="21914ebcf47a671f97437857ac2a3f64d6dca83ba38fcaeb616d7de5af787dc7"
Oct 30 00:57:21.600882 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:57:21.600854    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21914ebcf47a671f97437857ac2a3f64d6dca83ba38fcaeb616d7de5af787dc7\": container with ID starting with 21914ebcf47a671f97437857ac2a3f64d6dca83ba38fcaeb616d7de5af787dc7 not found: ID does not exist" containerID="21914ebcf47a671f97437857ac2a3f64d6dca83ba38fcaeb616d7de5af787dc7"
Oct 30 00:57:21.600956 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:21.600895    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21914ebcf47a671f97437857ac2a3f64d6dca83ba38fcaeb616d7de5af787dc7"} err="failed to get container status \"21914ebcf47a671f97437857ac2a3f64d6dca83ba38fcaeb616d7de5af787dc7\": rpc error: code = NotFound desc = could not find container \"21914ebcf47a671f97437857ac2a3f64d6dca83ba38fcaeb616d7de5af787dc7\": container with ID starting with 21914ebcf47a671f97437857ac2a3f64d6dca83ba38fcaeb616d7de5af787dc7 not found: ID does not exist"
Oct 30 00:57:21.640258 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:21.640177    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e207c138-6b79-4a2f-8987-0ecb06fc5503-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:57:21.640258 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:21.640217    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxdkv\" (UniqueName: \"kubernetes.io/projected/e207c138-6b79-4a2f-8987-0ecb06fc5503-kube-api-access-jxdkv\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:57:21.640258 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:21.640241    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e207c138-6b79-4a2f-8987-0ecb06fc5503-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:57:21.640542 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:21.640508    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw"]
Oct 30 00:57:21.646413 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:21.646388    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-02fbf5-6578df4d88-2pnlw"]
Oct 30 00:57:21.742519 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:21.742484    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e207c138-6b79-4a2f-8987-0ecb06fc5503" path="/var/lib/kubelet/pods/e207c138-6b79-4a2f-8987-0ecb06fc5503/volumes"
Oct 30 00:57:25.576655 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:25.576626    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2"
Oct 30 00:57:26.512972 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:57:26.512934    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:57:33.007741 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:33.007698    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf"]
Oct 30 00:57:33.009075 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:33.009048    4393 cpu_manager.go:414] "containerMap: removing container" podUID="e207c138-6b79-4a2f-8987-0ecb06fc5503" containerName="maas-api"
Oct 30 00:57:33.009075 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:33.009072    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e207c138-6b79-4a2f-8987-0ecb06fc5503" containerName="maas-api"
Oct 30 00:57:33.009343 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:33.009259    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="e207c138-6b79-4a2f-8987-0ecb06fc5503" containerName="maas-api"
Oct 30 00:57:33.014081 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:33.014057    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf"
Oct 30 00:57:33.030991 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:33.030965    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf"]
Oct 30 00:57:33.036269 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:33.036222    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b3bdf9\"/\"kube-root-ca.crt\""
Oct 30 00:57:33.040819 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:33.036380    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b3bdf9\"/\"test-model-b3bdf9-kserve-self-signed-certs\""
Oct 30 00:57:33.040819 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:33.036484    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b3bdf9\"/\"openshift-service-ca.crt\""
Oct 30 00:57:33.040819 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:33.039870    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b3bdf9\"/\"default-dockercfg-nhg7t\""
Oct 30 00:57:33.102332 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:33.102300    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a0449f-5a15-4e71-83aa-49d886d97a79-tls-certs\") pod \"test-model-b3bdf9-kserve-776bf69f7f-xgstf\" (UID: \"62a0449f-5a15-4e71-83aa-49d886d97a79\") " pod="ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf"
Oct 30 00:57:33.102332 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:33.102333    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a0449f-5a15-4e71-83aa-49d886d97a79-model-cache\") pod \"test-model-b3bdf9-kserve-776bf69f7f-xgstf\" (UID: \"62a0449f-5a15-4e71-83aa-49d886d97a79\") " pod="ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf"
Oct 30 00:57:33.102594 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:33.102422    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-988kk\" (UniqueName: \"kubernetes.io/projected/62a0449f-5a15-4e71-83aa-49d886d97a79-kube-api-access-988kk\") pod \"test-model-b3bdf9-kserve-776bf69f7f-xgstf\" (UID: \"62a0449f-5a15-4e71-83aa-49d886d97a79\") " pod="ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf"
Oct 30 00:57:33.102594 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:33.102514    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a0449f-5a15-4e71-83aa-49d886d97a79-dshm\") pod \"test-model-b3bdf9-kserve-776bf69f7f-xgstf\" (UID: \"62a0449f-5a15-4e71-83aa-49d886d97a79\") " pod="ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf"
Oct 30 00:57:33.102594 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:33.102555    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a0449f-5a15-4e71-83aa-49d886d97a79-tmp-dir\") pod \"test-model-b3bdf9-kserve-776bf69f7f-xgstf\" (UID: \"62a0449f-5a15-4e71-83aa-49d886d97a79\") " pod="ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf"
Oct 30 00:57:33.102751 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:33.102660    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a0449f-5a15-4e71-83aa-49d886d97a79-home\") pod \"test-model-b3bdf9-kserve-776bf69f7f-xgstf\" (UID: \"62a0449f-5a15-4e71-83aa-49d886d97a79\") " pod="ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf"
Oct 30 00:57:33.204542 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:33.204502    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a0449f-5a15-4e71-83aa-49d886d97a79-home\") pod \"test-model-b3bdf9-kserve-776bf69f7f-xgstf\" (UID: \"62a0449f-5a15-4e71-83aa-49d886d97a79\") " pod="ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf"
Oct 30 00:57:33.204756 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:33.204731    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a0449f-5a15-4e71-83aa-49d886d97a79-tls-certs\") pod \"test-model-b3bdf9-kserve-776bf69f7f-xgstf\" (UID: \"62a0449f-5a15-4e71-83aa-49d886d97a79\") " pod="ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf"
Oct 30 00:57:33.204890 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:33.204761    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a0449f-5a15-4e71-83aa-49d886d97a79-model-cache\") pod \"test-model-b3bdf9-kserve-776bf69f7f-xgstf\" (UID: \"62a0449f-5a15-4e71-83aa-49d886d97a79\") " pod="ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf"
Oct 30 00:57:33.204890 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:33.204865    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-988kk\" (UniqueName: \"kubernetes.io/projected/62a0449f-5a15-4e71-83aa-49d886d97a79-kube-api-access-988kk\") pod \"test-model-b3bdf9-kserve-776bf69f7f-xgstf\" (UID: \"62a0449f-5a15-4e71-83aa-49d886d97a79\") " pod="ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf"
Oct 30 00:57:33.205003 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:33.204935    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a0449f-5a15-4e71-83aa-49d886d97a79-dshm\") pod \"test-model-b3bdf9-kserve-776bf69f7f-xgstf\" (UID: \"62a0449f-5a15-4e71-83aa-49d886d97a79\") " pod="ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf"
Oct 30 00:57:33.205003 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:33.204942    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a0449f-5a15-4e71-83aa-49d886d97a79-home\") pod \"test-model-b3bdf9-kserve-776bf69f7f-xgstf\" (UID: \"62a0449f-5a15-4e71-83aa-49d886d97a79\") " pod="ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf"
Oct 30 00:57:33.205003 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:33.204981    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a0449f-5a15-4e71-83aa-49d886d97a79-tmp-dir\") pod \"test-model-b3bdf9-kserve-776bf69f7f-xgstf\" (UID: \"62a0449f-5a15-4e71-83aa-49d886d97a79\") " pod="ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf"
Oct 30 00:57:33.205209 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:33.205181    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a0449f-5a15-4e71-83aa-49d886d97a79-model-cache\") pod \"test-model-b3bdf9-kserve-776bf69f7f-xgstf\" (UID: \"62a0449f-5a15-4e71-83aa-49d886d97a79\") " pod="ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf"
Oct 30 00:57:33.205326 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:33.205307    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a0449f-5a15-4e71-83aa-49d886d97a79-tmp-dir\") pod \"test-model-b3bdf9-kserve-776bf69f7f-xgstf\" (UID: \"62a0449f-5a15-4e71-83aa-49d886d97a79\") " pod="ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf"
Oct 30 00:57:33.207337 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:33.207308    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a0449f-5a15-4e71-83aa-49d886d97a79-dshm\") pod \"test-model-b3bdf9-kserve-776bf69f7f-xgstf\" (UID: \"62a0449f-5a15-4e71-83aa-49d886d97a79\") " pod="ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf"
Oct 30 00:57:33.207452 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:33.207433    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a0449f-5a15-4e71-83aa-49d886d97a79-tls-certs\") pod \"test-model-b3bdf9-kserve-776bf69f7f-xgstf\" (UID: \"62a0449f-5a15-4e71-83aa-49d886d97a79\") " pod="ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf"
Oct 30 00:57:33.227552 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:33.227521    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-988kk\" (UniqueName: \"kubernetes.io/projected/62a0449f-5a15-4e71-83aa-49d886d97a79-kube-api-access-988kk\") pod \"test-model-b3bdf9-kserve-776bf69f7f-xgstf\" (UID: \"62a0449f-5a15-4e71-83aa-49d886d97a79\") " pod="ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf"
Oct 30 00:57:33.338854 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:33.337990    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf"
Oct 30 00:57:33.735519 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:33.735488    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf"]
Oct 30 00:57:33.737195 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:57:33.737139    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62a0449f_5a15_4e71_83aa_49d886d97a79.slice/crio-242e6f0e09e993ce87b7cbf233190fb16d9f91c470a5d3419458493616a3719d WatchSource:0}: Error finding container 242e6f0e09e993ce87b7cbf233190fb16d9f91c470a5d3419458493616a3719d: Status 404 returned error can't find the container with id 242e6f0e09e993ce87b7cbf233190fb16d9f91c470a5d3419458493616a3719d
Oct 30 00:57:34.755043 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:34.755009    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf" event={"ID":"62a0449f-5a15-4e71-83aa-49d886d97a79","Type":"ContainerStarted","Data":"8263a7846869e6db84446fcca183d0294e5c6906fd9bb82a5469619ed1a87097"}
Oct 30 00:57:34.755449 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:34.755054    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf" event={"ID":"62a0449f-5a15-4e71-83aa-49d886d97a79","Type":"ContainerStarted","Data":"242e6f0e09e993ce87b7cbf233190fb16d9f91c470a5d3419458493616a3719d"}
Oct 30 00:57:34.801405 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:34.801349    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf" podStartSLOduration=2.8013339569999998 podStartE2EDuration="2.801333957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:34.797594021 +0000 UTC m=+1545.185270409" watchObservedRunningTime="2026-10-30 00:57:34.801333957 +0000 UTC m=+1545.189010349"
Oct 30 00:57:35.339297 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:35.339264    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf"
Oct 30 00:57:40.975612 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:40.975575    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc"]
Oct 30 00:57:40.986412 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:40.986386    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc"
Oct 30 00:57:40.990686 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:40.990654    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc"]
Oct 30 00:57:41.006378 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:41.006346    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77782b40-06f9-4ba1-8c0b-fc787ee19a9b-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc\" (UID: \"77782b40-06f9-4ba1-8c0b-fc787ee19a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc"
Oct 30 00:57:41.006537 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:41.006402    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77782b40-06f9-4ba1-8c0b-fc787ee19a9b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc\" (UID: \"77782b40-06f9-4ba1-8c0b-fc787ee19a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc"
Oct 30 00:57:41.006537 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:41.006448    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77782b40-06f9-4ba1-8c0b-fc787ee19a9b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc\" (UID: \"77782b40-06f9-4ba1-8c0b-fc787ee19a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc"
Oct 30 00:57:41.006662 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:41.006626    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m86x\" (UniqueName: \"kubernetes.io/projected/77782b40-06f9-4ba1-8c0b-fc787ee19a9b-kube-api-access-9m86x\") pod \"maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc\" (UID: \"77782b40-06f9-4ba1-8c0b-fc787ee19a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc"
Oct 30 00:57:41.009109 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:41.009085    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-48df96-serving-cert\""
Oct 30 00:57:41.108825 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:41.108793    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77782b40-06f9-4ba1-8c0b-fc787ee19a9b-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc\" (UID: \"77782b40-06f9-4ba1-8c0b-fc787ee19a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc"
Oct 30 00:57:41.109580 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:41.109549    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77782b40-06f9-4ba1-8c0b-fc787ee19a9b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc\" (UID: \"77782b40-06f9-4ba1-8c0b-fc787ee19a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc"
Oct 30 00:57:41.109827 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:41.109807    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77782b40-06f9-4ba1-8c0b-fc787ee19a9b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc\" (UID: \"77782b40-06f9-4ba1-8c0b-fc787ee19a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc"
Oct 30 00:57:41.110135 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:41.110113    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m86x\" (UniqueName: \"kubernetes.io/projected/77782b40-06f9-4ba1-8c0b-fc787ee19a9b-kube-api-access-9m86x\") pod \"maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc\" (UID: \"77782b40-06f9-4ba1-8c0b-fc787ee19a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc"
Oct 30 00:57:41.110486 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:41.110459    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77782b40-06f9-4ba1-8c0b-fc787ee19a9b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc\" (UID: \"77782b40-06f9-4ba1-8c0b-fc787ee19a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc"
Oct 30 00:57:41.110583 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:57:41.110568    4393 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-48df96-serving-cert: secret "maas-api-e2e-shared-a-w1-48df96-serving-cert" not found
Oct 30 00:57:41.110665 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:57:41.110651    4393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77782b40-06f9-4ba1-8c0b-fc787ee19a9b-maas-api-tls podName:77782b40-06f9-4ba1-8c0b-fc787ee19a9b nodeName:}" failed. No retries permitted until 2026-10-30 00:57:41.610630993 +0000 UTC m=+1551.998307363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/77782b40-06f9-4ba1-8c0b-fc787ee19a9b-maas-api-tls") pod "maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc" (UID: "77782b40-06f9-4ba1-8c0b-fc787ee19a9b") : secret "maas-api-e2e-shared-a-w1-48df96-serving-cert" not found
Oct 30 00:57:41.128449 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:41.128323    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77782b40-06f9-4ba1-8c0b-fc787ee19a9b-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc\" (UID: \"77782b40-06f9-4ba1-8c0b-fc787ee19a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc"
Oct 30 00:57:41.133954 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:41.133932    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m86x\" (UniqueName: \"kubernetes.io/projected/77782b40-06f9-4ba1-8c0b-fc787ee19a9b-kube-api-access-9m86x\") pod \"maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc\" (UID: \"77782b40-06f9-4ba1-8c0b-fc787ee19a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc"
Oct 30 00:57:41.621956 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:41.621923    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77782b40-06f9-4ba1-8c0b-fc787ee19a9b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc\" (UID: \"77782b40-06f9-4ba1-8c0b-fc787ee19a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc"
Oct 30 00:57:41.624487 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:41.624467    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77782b40-06f9-4ba1-8c0b-fc787ee19a9b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc\" (UID: \"77782b40-06f9-4ba1-8c0b-fc787ee19a9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc"
Oct 30 00:57:41.910538 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:41.910450    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc"
Oct 30 00:57:42.218969 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:57:42.218930    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77782b40_06f9_4ba1_8c0b_fc787ee19a9b.slice/crio-34b0342bd32f7f445d0ee9e420bd12baa51e67ff291f686a33d5c7687373742e WatchSource:0}: Error finding container 34b0342bd32f7f445d0ee9e420bd12baa51e67ff291f686a33d5c7687373742e: Status 404 returned error can't find the container with id 34b0342bd32f7f445d0ee9e420bd12baa51e67ff291f686a33d5c7687373742e
Oct 30 00:57:42.219627 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:42.219548    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc"]
Oct 30 00:57:42.860442 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:42.860399    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc" event={"ID":"77782b40-06f9-4ba1-8c0b-fc787ee19a9b","Type":"ContainerStarted","Data":"0f088797ac0aaa0aa87dbc96708e36922ccedfd7ed6f5a9554d9fc01b04af4dd"}
Oct 30 00:57:42.860648 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:42.860473    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc" event={"ID":"77782b40-06f9-4ba1-8c0b-fc787ee19a9b","Type":"ContainerStarted","Data":"34b0342bd32f7f445d0ee9e420bd12baa51e67ff291f686a33d5c7687373742e"}
Oct 30 00:57:42.860648 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:42.860508    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc"
Oct 30 00:57:42.900834 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:42.900711    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc" podStartSLOduration=2.474013817 podStartE2EDuration="2.900696623s" totalImagesPullingTime="426.682806ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:57:42.900332092 +0000 UTC m=+1553.288008481" watchObservedRunningTime="2026-10-30 00:57:42.900696623 +0000 UTC m=+1553.288373011"
Oct 30 00:57:43.339050 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:43.338952    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf"
Oct 30 00:57:43.351414 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:43.351388    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf"
Oct 30 00:57:43.881102 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:43.881075    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf"
Oct 30 00:57:48.876328 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:48.876299    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc"
Oct 30 00:57:56.564240 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:57:56.564203    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:57:58.631946 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:58.631916    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg"]
Oct 30 00:57:58.638541 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:58.638517    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg"
Oct 30 00:57:58.658352 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:58.658326    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg"]
Oct 30 00:57:58.664946 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:58.664616    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-51b95e-serving-cert\""
Oct 30 00:57:58.707161 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:58.707126    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84c761db-2eb0-445d-b6e6-5c22476c834a-maas-api-tls\") pod \"maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg\" (UID: \"84c761db-2eb0-445d-b6e6-5c22476c834a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg"
Oct 30 00:57:58.707331 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:58.707259    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84c761db-2eb0-445d-b6e6-5c22476c834a-metrics-certs\") pod \"maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg\" (UID: \"84c761db-2eb0-445d-b6e6-5c22476c834a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg"
Oct 30 00:57:58.707331 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:58.707309    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84c761db-2eb0-445d-b6e6-5c22476c834a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg\" (UID: \"84c761db-2eb0-445d-b6e6-5c22476c834a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg"
Oct 30 00:57:58.707405 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:58.707357    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg87w\" (UniqueName: \"kubernetes.io/projected/84c761db-2eb0-445d-b6e6-5c22476c834a-kube-api-access-lg87w\") pod \"maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg\" (UID: \"84c761db-2eb0-445d-b6e6-5c22476c834a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg"
Oct 30 00:57:58.809097 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:58.809061    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84c761db-2eb0-445d-b6e6-5c22476c834a-maas-api-tls\") pod \"maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg\" (UID: \"84c761db-2eb0-445d-b6e6-5c22476c834a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg"
Oct 30 00:57:58.809287 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:58.809140    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84c761db-2eb0-445d-b6e6-5c22476c834a-metrics-certs\") pod \"maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg\" (UID: \"84c761db-2eb0-445d-b6e6-5c22476c834a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg"
Oct 30 00:57:58.809287 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:58.809162    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84c761db-2eb0-445d-b6e6-5c22476c834a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg\" (UID: \"84c761db-2eb0-445d-b6e6-5c22476c834a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg"
Oct 30 00:57:58.809287 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:58.809196    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg87w\" (UniqueName: \"kubernetes.io/projected/84c761db-2eb0-445d-b6e6-5c22476c834a-kube-api-access-lg87w\") pod \"maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg\" (UID: \"84c761db-2eb0-445d-b6e6-5c22476c834a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg"
Oct 30 00:57:58.809914 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:58.809891    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84c761db-2eb0-445d-b6e6-5c22476c834a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg\" (UID: \"84c761db-2eb0-445d-b6e6-5c22476c834a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg"
Oct 30 00:57:58.811810 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:58.811769    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84c761db-2eb0-445d-b6e6-5c22476c834a-maas-api-tls\") pod \"maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg\" (UID: \"84c761db-2eb0-445d-b6e6-5c22476c834a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg"
Oct 30 00:57:58.811908 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:58.811849    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84c761db-2eb0-445d-b6e6-5c22476c834a-metrics-certs\") pod \"maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg\" (UID: \"84c761db-2eb0-445d-b6e6-5c22476c834a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg"
Oct 30 00:57:58.832519 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:58.832493    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg87w\" (UniqueName: \"kubernetes.io/projected/84c761db-2eb0-445d-b6e6-5c22476c834a-kube-api-access-lg87w\") pod \"maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg\" (UID: \"84c761db-2eb0-445d-b6e6-5c22476c834a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg"
Oct 30 00:57:58.960423 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:58.960323    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg"
Oct 30 00:57:59.367293 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:57:59.367256    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg"]
Oct 30 00:58:00.218751 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:00.218712    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg" event={"ID":"84c761db-2eb0-445d-b6e6-5c22476c834a","Type":"ContainerStarted","Data":"38f67ea97bb4a56e8c1ddfb8bdf15f332ce2552f503fe8d85daae6289b780e30"}
Oct 30 00:58:01.228950 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:01.228850    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg" event={"ID":"84c761db-2eb0-445d-b6e6-5c22476c834a","Type":"ContainerStarted","Data":"1936a931e8a5df6c77370fb7bbe38df1eb606bcb99851f840f9d6addcedaa398"}
Oct 30 00:58:01.229409 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:01.229048    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg"
Oct 30 00:58:01.312750 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:01.312689    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg" podStartSLOduration=2.808834109 podStartE2EDuration="3.312664479s" totalImagesPullingTime="503.83037ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:57:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:01.287306289 +0000 UTC m=+1571.674982680" watchObservedRunningTime="2026-10-30 00:58:01.312664479 +0000 UTC m=+1571.700340882"
Oct 30 00:58:07.242949 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:07.242920    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg"
Oct 30 00:58:13.027632 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:13.027601    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg"]
Oct 30 00:58:13.032450 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:13.032428    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg"
Oct 30 00:58:13.054087 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:13.054059    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg"]
Oct 30 00:58:13.060881 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:13.060857    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-8d2a70-serving-cert\""
Oct 30 00:58:13.187682 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:13.187649    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98869821-d2ad-490b-b71b-ea9d17af6327-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg\" (UID: \"98869821-d2ad-490b-b71b-ea9d17af6327\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg"
Oct 30 00:58:13.187873 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:13.187698    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98869821-d2ad-490b-b71b-ea9d17af6327-metrics-certs\") pod \"maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg\" (UID: \"98869821-d2ad-490b-b71b-ea9d17af6327\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg"
Oct 30 00:58:13.187873 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:13.187828    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r87tk\" (UniqueName: \"kubernetes.io/projected/98869821-d2ad-490b-b71b-ea9d17af6327-kube-api-access-r87tk\") pod \"maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg\" (UID: \"98869821-d2ad-490b-b71b-ea9d17af6327\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg"
Oct 30 00:58:13.188128 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:13.188108    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98869821-d2ad-490b-b71b-ea9d17af6327-maas-api-tls\") pod \"maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg\" (UID: \"98869821-d2ad-490b-b71b-ea9d17af6327\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg"
Oct 30 00:58:13.302434 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:13.293419    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98869821-d2ad-490b-b71b-ea9d17af6327-maas-api-tls\") pod \"maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg\" (UID: \"98869821-d2ad-490b-b71b-ea9d17af6327\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg"
Oct 30 00:58:13.303394 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:13.303355    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98869821-d2ad-490b-b71b-ea9d17af6327-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg\" (UID: \"98869821-d2ad-490b-b71b-ea9d17af6327\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg"
Oct 30 00:58:13.304769 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:13.304742    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98869821-d2ad-490b-b71b-ea9d17af6327-metrics-certs\") pod \"maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg\" (UID: \"98869821-d2ad-490b-b71b-ea9d17af6327\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg"
Oct 30 00:58:13.305474 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:13.305450    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r87tk\" (UniqueName: \"kubernetes.io/projected/98869821-d2ad-490b-b71b-ea9d17af6327-kube-api-access-r87tk\") pod \"maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg\" (UID: \"98869821-d2ad-490b-b71b-ea9d17af6327\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg"
Oct 30 00:58:13.306503 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:13.304128    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98869821-d2ad-490b-b71b-ea9d17af6327-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg\" (UID: \"98869821-d2ad-490b-b71b-ea9d17af6327\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg"
Oct 30 00:58:13.350936 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:13.347895    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98869821-d2ad-490b-b71b-ea9d17af6327-maas-api-tls\") pod \"maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg\" (UID: \"98869821-d2ad-490b-b71b-ea9d17af6327\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg"
Oct 30 00:58:13.378860 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:13.374477    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r87tk\" (UniqueName: \"kubernetes.io/projected/98869821-d2ad-490b-b71b-ea9d17af6327-kube-api-access-r87tk\") pod \"maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg\" (UID: \"98869821-d2ad-490b-b71b-ea9d17af6327\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg"
Oct 30 00:58:13.379287 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:13.379203    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98869821-d2ad-490b-b71b-ea9d17af6327-metrics-certs\") pod \"maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg\" (UID: \"98869821-d2ad-490b-b71b-ea9d17af6327\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg"
Oct 30 00:58:13.648642 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:13.648611    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg"
Oct 30 00:58:13.956514 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:58:13.956482    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98869821_d2ad_490b_b71b_ea9d17af6327.slice/crio-3287a585e5db03c2bf597dde9d3b666ceb218f3c3475749766517100e4a58950 WatchSource:0}: Error finding container 3287a585e5db03c2bf597dde9d3b666ceb218f3c3475749766517100e4a58950: Status 404 returned error can't find the container with id 3287a585e5db03c2bf597dde9d3b666ceb218f3c3475749766517100e4a58950
Oct 30 00:58:13.957179 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:13.957156    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg"]
Oct 30 00:58:14.418802 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:14.417853    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg" event={"ID":"98869821-d2ad-490b-b71b-ea9d17af6327","Type":"ContainerStarted","Data":"3287a585e5db03c2bf597dde9d3b666ceb218f3c3475749766517100e4a58950"}
Oct 30 00:58:15.431874 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:15.431819    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg" event={"ID":"98869821-d2ad-490b-b71b-ea9d17af6327","Type":"ContainerStarted","Data":"e9bb9da24f9ddea11c331518ac564de0900a6588d3405cd7d8ca5aeb324322be"}
Oct 30 00:58:15.433425 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:15.433397    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg"
Oct 30 00:58:15.476304 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:15.476255    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg" podStartSLOduration=2.00601531 podStartE2EDuration="2.476240656s" totalImagesPullingTime="470.225346ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:15.473221961 +0000 UTC m=+1585.860898352" watchObservedRunningTime="2026-10-30 00:58:15.476240656 +0000 UTC m=+1585.863917044"
Oct 30 00:58:22.453616 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:22.453584    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg"
Oct 30 00:58:26.512736 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:58:26.512693    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:58:40.951298 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:40.951264    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct"]
Oct 30 00:58:40.957067 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:40.957041    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct"
Oct 30 00:58:40.971527 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:40.971500    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-03d892-serving-cert\""
Oct 30 00:58:40.976573 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:40.976545    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct"]
Oct 30 00:58:41.022159 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:41.022110    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/211b2ad5-c753-4671-9be9-96b8f39aa549-metrics-certs\") pod \"maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct\" (UID: \"211b2ad5-c753-4671-9be9-96b8f39aa549\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct"
Oct 30 00:58:41.022328 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:41.022161    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcc4j\" (UniqueName: \"kubernetes.io/projected/211b2ad5-c753-4671-9be9-96b8f39aa549-kube-api-access-tcc4j\") pod \"maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct\" (UID: \"211b2ad5-c753-4671-9be9-96b8f39aa549\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct"
Oct 30 00:58:41.022328 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:41.022266    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/211b2ad5-c753-4671-9be9-96b8f39aa549-openshift-service-ca\") pod \"maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct\" (UID: \"211b2ad5-c753-4671-9be9-96b8f39aa549\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct"
Oct 30 00:58:41.022456 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:41.022438    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/211b2ad5-c753-4671-9be9-96b8f39aa549-maas-api-tls\") pod \"maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct\" (UID: \"211b2ad5-c753-4671-9be9-96b8f39aa549\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct"
Oct 30 00:58:41.178802 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:41.136953    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/211b2ad5-c753-4671-9be9-96b8f39aa549-metrics-certs\") pod \"maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct\" (UID: \"211b2ad5-c753-4671-9be9-96b8f39aa549\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct"
Oct 30 00:58:41.179095 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:41.179063    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcc4j\" (UniqueName: \"kubernetes.io/projected/211b2ad5-c753-4671-9be9-96b8f39aa549-kube-api-access-tcc4j\") pod \"maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct\" (UID: \"211b2ad5-c753-4671-9be9-96b8f39aa549\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct"
Oct 30 00:58:41.179273 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:41.179256    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/211b2ad5-c753-4671-9be9-96b8f39aa549-openshift-service-ca\") pod \"maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct\" (UID: \"211b2ad5-c753-4671-9be9-96b8f39aa549\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct"
Oct 30 00:58:41.179628 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:41.179598    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/211b2ad5-c753-4671-9be9-96b8f39aa549-maas-api-tls\") pod \"maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct\" (UID: \"211b2ad5-c753-4671-9be9-96b8f39aa549\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct"
Oct 30 00:58:41.186719 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:41.151490    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/211b2ad5-c753-4671-9be9-96b8f39aa549-metrics-certs\") pod \"maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct\" (UID: \"211b2ad5-c753-4671-9be9-96b8f39aa549\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct"
Oct 30 00:58:41.186885 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:41.184162    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/211b2ad5-c753-4671-9be9-96b8f39aa549-openshift-service-ca\") pod \"maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct\" (UID: \"211b2ad5-c753-4671-9be9-96b8f39aa549\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct"
Oct 30 00:58:41.186885 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:58:41.185232    4393 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-03d892-serving-cert: secret "maas-api-e2e-inf-b-03d892-serving-cert" not found
Oct 30 00:58:41.187197 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:58:41.187154    4393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/211b2ad5-c753-4671-9be9-96b8f39aa549-maas-api-tls podName:211b2ad5-c753-4671-9be9-96b8f39aa549 nodeName:}" failed. No retries permitted until 2026-10-30 00:58:41.687124742 +0000 UTC m=+1612.074801113 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/211b2ad5-c753-4671-9be9-96b8f39aa549-maas-api-tls") pod "maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct" (UID: "211b2ad5-c753-4671-9be9-96b8f39aa549") : secret "maas-api-e2e-inf-b-03d892-serving-cert" not found
Oct 30 00:58:41.215645 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:41.215565    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcc4j\" (UniqueName: \"kubernetes.io/projected/211b2ad5-c753-4671-9be9-96b8f39aa549-kube-api-access-tcc4j\") pod \"maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct\" (UID: \"211b2ad5-c753-4671-9be9-96b8f39aa549\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct"
Oct 30 00:58:41.696008 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:41.695974    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/211b2ad5-c753-4671-9be9-96b8f39aa549-maas-api-tls\") pod \"maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct\" (UID: \"211b2ad5-c753-4671-9be9-96b8f39aa549\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct"
Oct 30 00:58:41.698657 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:41.698631    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/211b2ad5-c753-4671-9be9-96b8f39aa549-maas-api-tls\") pod \"maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct\" (UID: \"211b2ad5-c753-4671-9be9-96b8f39aa549\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct"
Oct 30 00:58:41.899559 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:41.899519    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct"
Oct 30 00:58:42.282273 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:58:42.282224    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod211b2ad5_c753_4671_9be9_96b8f39aa549.slice/crio-c229b369d88c3205e121110d705a90b397af32db7288dd53440281c0a8a518c1 WatchSource:0}: Error finding container c229b369d88c3205e121110d705a90b397af32db7288dd53440281c0a8a518c1: Status 404 returned error can't find the container with id c229b369d88c3205e121110d705a90b397af32db7288dd53440281c0a8a518c1
Oct 30 00:58:42.282916 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:42.282768    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct"]
Oct 30 00:58:42.828946 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:42.828903    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct" event={"ID":"211b2ad5-c753-4671-9be9-96b8f39aa549","Type":"ContainerStarted","Data":"8dd176dc9b88d239bbdffc02595fbb27fcbf07d7fb830d33e2db5a21e1424de0"}
Oct 30 00:58:42.829159 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:42.828973    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct"
Oct 30 00:58:42.829159 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:42.828995    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct" event={"ID":"211b2ad5-c753-4671-9be9-96b8f39aa549","Type":"ContainerStarted","Data":"c229b369d88c3205e121110d705a90b397af32db7288dd53440281c0a8a518c1"}
Oct 30 00:58:42.871343 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:42.871282    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct" podStartSLOduration=2.552625545 podStartE2EDuration="2.871263043s" totalImagesPullingTime="318.637498ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:42.869105444 +0000 UTC m=+1613.256781835" watchObservedRunningTime="2026-10-30 00:58:42.871263043 +0000 UTC m=+1613.258939433"
Oct 30 00:58:44.835700 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:44.835664    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p"]
Oct 30 00:58:44.841738 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:44.841714    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p"
Oct 30 00:58:44.878266 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:44.878235    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-84b89f-serving-cert\""
Oct 30 00:58:44.904250 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:44.904217    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ace5c083-c41b-46bd-9a0e-5545c4e141ea-maas-api-tls\") pod \"maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p\" (UID: \"ace5c083-c41b-46bd-9a0e-5545c4e141ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p"
Oct 30 00:58:44.904433 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:44.904325    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p74h\" (UniqueName: \"kubernetes.io/projected/ace5c083-c41b-46bd-9a0e-5545c4e141ea-kube-api-access-8p74h\") pod \"maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p\" (UID: \"ace5c083-c41b-46bd-9a0e-5545c4e141ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p"
Oct 30 00:58:44.904515 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:44.904430    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ace5c083-c41b-46bd-9a0e-5545c4e141ea-openshift-service-ca\") pod \"maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p\" (UID: \"ace5c083-c41b-46bd-9a0e-5545c4e141ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p"
Oct 30 00:58:44.904624 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:44.904605    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ace5c083-c41b-46bd-9a0e-5545c4e141ea-metrics-certs\") pod \"maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p\" (UID: \"ace5c083-c41b-46bd-9a0e-5545c4e141ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p"
Oct 30 00:58:44.913168 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:44.913139    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p"]
Oct 30 00:58:45.006011 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:45.005977    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p74h\" (UniqueName: \"kubernetes.io/projected/ace5c083-c41b-46bd-9a0e-5545c4e141ea-kube-api-access-8p74h\") pod \"maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p\" (UID: \"ace5c083-c41b-46bd-9a0e-5545c4e141ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p"
Oct 30 00:58:45.006222 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:45.006063    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ace5c083-c41b-46bd-9a0e-5545c4e141ea-openshift-service-ca\") pod \"maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p\" (UID: \"ace5c083-c41b-46bd-9a0e-5545c4e141ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p"
Oct 30 00:58:45.006222 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:45.006181    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ace5c083-c41b-46bd-9a0e-5545c4e141ea-metrics-certs\") pod \"maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p\" (UID: \"ace5c083-c41b-46bd-9a0e-5545c4e141ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p"
Oct 30 00:58:45.006346 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:45.006268    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ace5c083-c41b-46bd-9a0e-5545c4e141ea-maas-api-tls\") pod \"maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p\" (UID: \"ace5c083-c41b-46bd-9a0e-5545c4e141ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p"
Oct 30 00:58:45.006820 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:45.006796    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ace5c083-c41b-46bd-9a0e-5545c4e141ea-openshift-service-ca\") pod \"maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p\" (UID: \"ace5c083-c41b-46bd-9a0e-5545c4e141ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p"
Oct 30 00:58:45.008987 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:45.008966    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ace5c083-c41b-46bd-9a0e-5545c4e141ea-metrics-certs\") pod \"maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p\" (UID: \"ace5c083-c41b-46bd-9a0e-5545c4e141ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p"
Oct 30 00:58:45.009083 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:45.009024    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ace5c083-c41b-46bd-9a0e-5545c4e141ea-maas-api-tls\") pod \"maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p\" (UID: \"ace5c083-c41b-46bd-9a0e-5545c4e141ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p"
Oct 30 00:58:45.031676 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:45.031650    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p74h\" (UniqueName: \"kubernetes.io/projected/ace5c083-c41b-46bd-9a0e-5545c4e141ea-kube-api-access-8p74h\") pod \"maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p\" (UID: \"ace5c083-c41b-46bd-9a0e-5545c4e141ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p"
Oct 30 00:58:45.158142 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:45.158106    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p"
Oct 30 00:58:45.491763 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:58:45.491730    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podace5c083_c41b_46bd_9a0e_5545c4e141ea.slice/crio-aa543d719f1cf478e820ebe69d2240e1a9e7543d7574602a5e4723d7a3af6a98 WatchSource:0}: Error finding container aa543d719f1cf478e820ebe69d2240e1a9e7543d7574602a5e4723d7a3af6a98: Status 404 returned error can't find the container with id aa543d719f1cf478e820ebe69d2240e1a9e7543d7574602a5e4723d7a3af6a98
Oct 30 00:58:45.492515 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:45.492485    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p"]
Oct 30 00:58:45.891324 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:45.891285    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p" event={"ID":"ace5c083-c41b-46bd-9a0e-5545c4e141ea","Type":"ContainerStarted","Data":"aa543d719f1cf478e820ebe69d2240e1a9e7543d7574602a5e4723d7a3af6a98"}
Oct 30 00:58:46.717158 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:46.717123    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"]
Oct 30 00:58:46.722336 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:46.722315    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"
Oct 30 00:58:46.754310 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:46.754279    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"]
Oct 30 00:58:46.810167 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:46.810130    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762e23fe-a544-49c1-a734-1c93e8bac082-workload-certs\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw\" (UID: \"762e23fe-a544-49c1-a734-1c93e8bac082\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"
Oct 30 00:58:46.810536 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:46.810502    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762e23fe-a544-49c1-a734-1c93e8bac082-istio-data\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw\" (UID: \"762e23fe-a544-49c1-a734-1c93e8bac082\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"
Oct 30 00:58:46.810705 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:46.810659    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762e23fe-a544-49c1-a734-1c93e8bac082-istio-podinfo\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw\" (UID: \"762e23fe-a544-49c1-a734-1c93e8bac082\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"
Oct 30 00:58:46.810705 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:46.810693    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762e23fe-a544-49c1-a734-1c93e8bac082-credential-socket\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw\" (UID: \"762e23fe-a544-49c1-a734-1c93e8bac082\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"
Oct 30 00:58:46.810872 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:46.810727    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762e23fe-a544-49c1-a734-1c93e8bac082-istio-envoy\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw\" (UID: \"762e23fe-a544-49c1-a734-1c93e8bac082\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"
Oct 30 00:58:46.810872 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:46.810757    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg779\" (UniqueName: \"kubernetes.io/projected/762e23fe-a544-49c1-a734-1c93e8bac082-kube-api-access-sg779\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw\" (UID: \"762e23fe-a544-49c1-a734-1c93e8bac082\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"
Oct 30 00:58:46.810996 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:46.810941    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/762e23fe-a544-49c1-a734-1c93e8bac082-istio-token\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw\" (UID: \"762e23fe-a544-49c1-a734-1c93e8bac082\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"
Oct 30 00:58:46.810996 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:46.810976    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762e23fe-a544-49c1-a734-1c93e8bac082-istiod-ca-cert\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw\" (UID: \"762e23fe-a544-49c1-a734-1c93e8bac082\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"
Oct 30 00:58:46.811259 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:46.811241    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762e23fe-a544-49c1-a734-1c93e8bac082-workload-socket\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw\" (UID: \"762e23fe-a544-49c1-a734-1c93e8bac082\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"
Oct 30 00:58:46.829950 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:46.829918    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8d2a70-openshift-default-dockercfg-vpjnj\""
Oct 30 00:58:46.901489 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:46.901450    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p" event={"ID":"ace5c083-c41b-46bd-9a0e-5545c4e141ea","Type":"ContainerStarted","Data":"fa9f3c0926f83a402ddd4bcb866337eac01a1395bc687e1e6009ac7d3be98498"}
Oct 30 00:58:46.901957 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:46.901559    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p"
Oct 30 00:58:46.912564 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:46.912531    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/762e23fe-a544-49c1-a734-1c93e8bac082-istio-token\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw\" (UID: \"762e23fe-a544-49c1-a734-1c93e8bac082\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"
Oct 30 00:58:46.912761 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:46.912595    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762e23fe-a544-49c1-a734-1c93e8bac082-istiod-ca-cert\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw\" (UID: \"762e23fe-a544-49c1-a734-1c93e8bac082\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"
Oct 30 00:58:46.912761 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:46.912687    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762e23fe-a544-49c1-a734-1c93e8bac082-workload-socket\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw\" (UID: \"762e23fe-a544-49c1-a734-1c93e8bac082\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"
Oct 30 00:58:46.913045 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:46.913023    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762e23fe-a544-49c1-a734-1c93e8bac082-workload-certs\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw\" (UID: \"762e23fe-a544-49c1-a734-1c93e8bac082\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"
Oct 30 00:58:46.913133 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:46.913069    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762e23fe-a544-49c1-a734-1c93e8bac082-istio-data\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw\" (UID: \"762e23fe-a544-49c1-a734-1c93e8bac082\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"
Oct 30 00:58:46.913189 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:46.913179    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762e23fe-a544-49c1-a734-1c93e8bac082-istio-podinfo\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw\" (UID: \"762e23fe-a544-49c1-a734-1c93e8bac082\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"
Oct 30 00:58:46.913240 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:46.913207    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762e23fe-a544-49c1-a734-1c93e8bac082-credential-socket\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw\" (UID: \"762e23fe-a544-49c1-a734-1c93e8bac082\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"
Oct 30 00:58:46.913301 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:46.913238    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762e23fe-a544-49c1-a734-1c93e8bac082-istio-envoy\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw\" (UID: \"762e23fe-a544-49c1-a734-1c93e8bac082\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"
Oct 30 00:58:46.913301 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:46.913268    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg779\" (UniqueName: \"kubernetes.io/projected/762e23fe-a544-49c1-a734-1c93e8bac082-kube-api-access-sg779\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw\" (UID: \"762e23fe-a544-49c1-a734-1c93e8bac082\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"
Oct 30 00:58:46.913720 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:46.913695    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762e23fe-a544-49c1-a734-1c93e8bac082-istiod-ca-cert\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw\" (UID: \"762e23fe-a544-49c1-a734-1c93e8bac082\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"
Oct 30 00:58:46.914006 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:46.913986    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762e23fe-a544-49c1-a734-1c93e8bac082-istio-data\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw\" (UID: \"762e23fe-a544-49c1-a734-1c93e8bac082\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"
Oct 30 00:58:46.914086 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:46.913984    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762e23fe-a544-49c1-a734-1c93e8bac082-workload-socket\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw\" (UID: \"762e23fe-a544-49c1-a734-1c93e8bac082\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"
Oct 30 00:58:46.914623 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:46.914585    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762e23fe-a544-49c1-a734-1c93e8bac082-workload-certs\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw\" (UID: \"762e23fe-a544-49c1-a734-1c93e8bac082\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"
Oct 30 00:58:46.914740 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:46.914721    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762e23fe-a544-49c1-a734-1c93e8bac082-credential-socket\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw\" (UID: \"762e23fe-a544-49c1-a734-1c93e8bac082\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"
Oct 30 00:58:46.917053 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:46.917032    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762e23fe-a544-49c1-a734-1c93e8bac082-istio-podinfo\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw\" (UID: \"762e23fe-a544-49c1-a734-1c93e8bac082\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"
Oct 30 00:58:46.917205 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:46.917180    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762e23fe-a544-49c1-a734-1c93e8bac082-istio-envoy\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw\" (UID: \"762e23fe-a544-49c1-a734-1c93e8bac082\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"
Oct 30 00:58:46.935336 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:46.935303    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/762e23fe-a544-49c1-a734-1c93e8bac082-istio-token\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw\" (UID: \"762e23fe-a544-49c1-a734-1c93e8bac082\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"
Oct 30 00:58:46.935511 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:46.935492    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg779\" (UniqueName: \"kubernetes.io/projected/762e23fe-a544-49c1-a734-1c93e8bac082-kube-api-access-sg779\") pod \"e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw\" (UID: \"762e23fe-a544-49c1-a734-1c93e8bac082\") " pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"
Oct 30 00:58:46.944757 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:46.944715    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p" podStartSLOduration=2.612668947 podStartE2EDuration="2.944701634s" totalImagesPullingTime="332.032687ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:46.942107328 +0000 UTC m=+1617.329783719" watchObservedRunningTime="2026-10-30 00:58:46.944701634 +0000 UTC m=+1617.332378023"
Oct 30 00:58:47.063901 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:47.063814    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"
Oct 30 00:58:47.484287 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:58:47.484243    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod762e23fe_a544_49c1_a734_1c93e8bac082.slice/crio-54fd75452dd897ce8160c4413ca0b021b662572f22267a6dc2939dd6b1582066 WatchSource:0}: Error finding container 54fd75452dd897ce8160c4413ca0b021b662572f22267a6dc2939dd6b1582066: Status 404 returned error can't find the container with id 54fd75452dd897ce8160c4413ca0b021b662572f22267a6dc2939dd6b1582066
Oct 30 00:58:47.485554 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:47.485532    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"]
Oct 30 00:58:47.487499 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:47.487467    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:58:47.487579 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:47.487532    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:58:47.487579 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:47.487563    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 00:58:47.915901 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:47.915869    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw" event={"ID":"762e23fe-a544-49c1-a734-1c93e8bac082","Type":"ContainerStarted","Data":"14266b28595a9771a400bc3f01c7907a46f2aed1779a72615e1620a7638d9706"}
Oct 30 00:58:47.916435 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:47.916404    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw" event={"ID":"762e23fe-a544-49c1-a734-1c93e8bac082","Type":"ContainerStarted","Data":"54fd75452dd897ce8160c4413ca0b021b662572f22267a6dc2939dd6b1582066"}
Oct 30 00:58:47.987308 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:47.987244    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw" podStartSLOduration=1.9872248940000001 podStartE2EDuration="1.987224894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:58:47.96996747 +0000 UTC m=+1618.357643861" watchObservedRunningTime="2026-10-30 00:58:47.987224894 +0000 UTC m=+1618.374901285"
Oct 30 00:58:48.065036 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:48.064996    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"
Oct 30 00:58:48.066941 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:48.066913    4393 patch_prober.go:29] interesting pod/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 00:58:48.067074 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:48.066968    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw" podUID="762e23fe-a544-49c1-a734-1c93e8bac082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 00:58:48.843721 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:48.843691    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct"
Oct 30 00:58:49.064614 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:49.064579    4393 patch_prober.go:29] interesting pod/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 00:58:49.064998 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:49.064628    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw" podUID="762e23fe-a544-49c1-a734-1c93e8bac082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 00:58:50.066171 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:50.066138    4393 patch_prober.go:29] interesting pod/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 00:58:50.066641 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:50.066196    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw" podUID="762e23fe-a544-49c1-a734-1c93e8bac082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 00:58:51.064826 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:51.064758    4393 patch_prober.go:29] interesting pod/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 00:58:51.065034 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:51.064852    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw" podUID="762e23fe-a544-49c1-a734-1c93e8bac082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 00:58:52.065909 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:52.065872    4393 patch_prober.go:29] interesting pod/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 00:58:52.066313 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:52.065934    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw" podUID="762e23fe-a544-49c1-a734-1c93e8bac082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 00:58:52.929746 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:52.929716    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p"
Oct 30 00:58:53.065041 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:53.065007    4393 patch_prober.go:29] interesting pod/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 00:58:53.065355 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:53.065318    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw" podUID="762e23fe-a544-49c1-a734-1c93e8bac082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 00:58:54.199553 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:54.199519    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"
Oct 30 00:58:54.200541 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:54.200503    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"
Oct 30 00:58:54.206059 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:58:54.206036    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"
Oct 30 00:58:56.513281 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:58:56.513247    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:59:26.513978 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:59:26.513939    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:59:37.814880 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:37.814769    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"]
Oct 30 00:59:37.815329 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:37.815050    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx" podUID="d42228bf-b9c4-4be5-bc94-ae937377708b" containerName="istio-proxy" containerID="cri-o://a3961aa54255c8969a872cc6c2795415c4ab2a2d6b034c9eb4f2b3f6bbda05e6" gracePeriod=30
Oct 30 00:59:38.232221 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:38.232182    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"]
Oct 30 00:59:38.232451 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:38.232424    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6" podUID="3b626471-54fd-4301-89fa-cc1738c35cce" containerName="istio-proxy" containerID="cri-o://f5ea02f1a8c1d0a0248db155606b648e531de07392eda5cdc00fc4edead745a8" gracePeriod=30
Oct 30 00:59:43.124418 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.124382    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"
Oct 30 00:59:43.251362 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.251337    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d42228bf-b9c4-4be5-bc94-ae937377708b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42228bf-b9c4-4be5-bc94-ae937377708b-istio-data\") pod \"d42228bf-b9c4-4be5-bc94-ae937377708b\" (UID: \"d42228bf-b9c4-4be5-bc94-ae937377708b\") "
Oct 30 00:59:43.251472 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.251378    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d42228bf-b9c4-4be5-bc94-ae937377708b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42228bf-b9c4-4be5-bc94-ae937377708b-workload-certs\") pod \"d42228bf-b9c4-4be5-bc94-ae937377708b\" (UID: \"d42228bf-b9c4-4be5-bc94-ae937377708b\") "
Oct 30 00:59:43.251526 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.251488    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d42228bf-b9c4-4be5-bc94-ae937377708b-kube-api-access-bfkzm\" (UniqueName: \"kubernetes.io/projected/d42228bf-b9c4-4be5-bc94-ae937377708b-kube-api-access-bfkzm\") pod \"d42228bf-b9c4-4be5-bc94-ae937377708b\" (UID: \"d42228bf-b9c4-4be5-bc94-ae937377708b\") "
Oct 30 00:59:43.251579 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.251531    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d42228bf-b9c4-4be5-bc94-ae937377708b-istio-token\" (UniqueName: \"kubernetes.io/projected/d42228bf-b9c4-4be5-bc94-ae937377708b-istio-token\") pod \"d42228bf-b9c4-4be5-bc94-ae937377708b\" (UID: \"d42228bf-b9c4-4be5-bc94-ae937377708b\") "
Oct 30 00:59:43.251658 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.251629    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42228bf-b9c4-4be5-bc94-ae937377708b-istio-data" pod "d42228bf-b9c4-4be5-bc94-ae937377708b" (UID: "d42228bf-b9c4-4be5-bc94-ae937377708b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:43.251658 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.251648    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42228bf-b9c4-4be5-bc94-ae937377708b-workload-certs" pod "d42228bf-b9c4-4be5-bc94-ae937377708b" (UID: "d42228bf-b9c4-4be5-bc94-ae937377708b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:43.251766 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.251672    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d42228bf-b9c4-4be5-bc94-ae937377708b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42228bf-b9c4-4be5-bc94-ae937377708b-istiod-ca-cert\") pod \"d42228bf-b9c4-4be5-bc94-ae937377708b\" (UID: \"d42228bf-b9c4-4be5-bc94-ae937377708b\") "
Oct 30 00:59:43.251766 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.251697    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d42228bf-b9c4-4be5-bc94-ae937377708b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42228bf-b9c4-4be5-bc94-ae937377708b-credential-socket\") pod \"d42228bf-b9c4-4be5-bc94-ae937377708b\" (UID: \"d42228bf-b9c4-4be5-bc94-ae937377708b\") "
Oct 30 00:59:43.251931 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.251767    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d42228bf-b9c4-4be5-bc94-ae937377708b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42228bf-b9c4-4be5-bc94-ae937377708b-workload-socket\") pod \"d42228bf-b9c4-4be5-bc94-ae937377708b\" (UID: \"d42228bf-b9c4-4be5-bc94-ae937377708b\") "
Oct 30 00:59:43.251931 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.251820    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d42228bf-b9c4-4be5-bc94-ae937377708b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42228bf-b9c4-4be5-bc94-ae937377708b-istio-envoy\") pod \"d42228bf-b9c4-4be5-bc94-ae937377708b\" (UID: \"d42228bf-b9c4-4be5-bc94-ae937377708b\") "
Oct 30 00:59:43.251931 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.251863    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d42228bf-b9c4-4be5-bc94-ae937377708b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42228bf-b9c4-4be5-bc94-ae937377708b-istio-podinfo\") pod \"d42228bf-b9c4-4be5-bc94-ae937377708b\" (UID: \"d42228bf-b9c4-4be5-bc94-ae937377708b\") "
Oct 30 00:59:43.252098 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.251986    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42228bf-b9c4-4be5-bc94-ae937377708b-credential-socket" pod "d42228bf-b9c4-4be5-bc94-ae937377708b" (UID: "d42228bf-b9c4-4be5-bc94-ae937377708b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:43.252098 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.252057    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42228bf-b9c4-4be5-bc94-ae937377708b-workload-socket" pod "d42228bf-b9c4-4be5-bc94-ae937377708b" (UID: "d42228bf-b9c4-4be5-bc94-ae937377708b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:43.252098 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.252090    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d42228bf-b9c4-4be5-bc94-ae937377708b-istiod-ca-cert" pod "d42228bf-b9c4-4be5-bc94-ae937377708b" (UID: "d42228bf-b9c4-4be5-bc94-ae937377708b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:59:43.252719 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.252697    4393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42228bf-b9c4-4be5-bc94-ae937377708b-credential-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.252719 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.252719    4393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42228bf-b9c4-4be5-bc94-ae937377708b-workload-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.252990 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.252732    4393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42228bf-b9c4-4be5-bc94-ae937377708b-istio-data\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.252990 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.252746    4393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42228bf-b9c4-4be5-bc94-ae937377708b-workload-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.252990 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.252762    4393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42228bf-b9c4-4be5-bc94-ae937377708b-istiod-ca-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.254329 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.254299    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42228bf-b9c4-4be5-bc94-ae937377708b-istio-envoy" pod "d42228bf-b9c4-4be5-bc94-ae937377708b" (UID: "d42228bf-b9c4-4be5-bc94-ae937377708b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:43.254501 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.254480    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d42228bf-b9c4-4be5-bc94-ae937377708b-kube-api-access-bfkzm" pod "d42228bf-b9c4-4be5-bc94-ae937377708b" (UID: "d42228bf-b9c4-4be5-bc94-ae937377708b"). InnerVolumeSpecName "kube-api-access-bfkzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:43.254913 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.254890    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d42228bf-b9c4-4be5-bc94-ae937377708b-istio-podinfo" pod "d42228bf-b9c4-4be5-bc94-ae937377708b" (UID: "d42228bf-b9c4-4be5-bc94-ae937377708b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:59:43.255189 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.255172    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d42228bf-b9c4-4be5-bc94-ae937377708b-istio-token" pod "d42228bf-b9c4-4be5-bc94-ae937377708b" (UID: "d42228bf-b9c4-4be5-bc94-ae937377708b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:43.372914 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.368850    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfkzm\" (UniqueName: \"kubernetes.io/projected/d42228bf-b9c4-4be5-bc94-ae937377708b-kube-api-access-bfkzm\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.372914 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.368879    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d42228bf-b9c4-4be5-bc94-ae937377708b-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.372914 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.368894    4393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42228bf-b9c4-4be5-bc94-ae937377708b-istio-envoy\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.372914 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.368907    4393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42228bf-b9c4-4be5-bc94-ae937377708b-istio-podinfo\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.563529 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.563505    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"
Oct 30 00:59:43.619117 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.619075    4393 generic.go:356] "Generic (PLEG): container finished" podID="d42228bf-b9c4-4be5-bc94-ae937377708b" containerID="a3961aa54255c8969a872cc6c2795415c4ab2a2d6b034c9eb4f2b3f6bbda05e6" exitCode=0
Oct 30 00:59:43.619363 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.619189    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx" event={"ID":"d42228bf-b9c4-4be5-bc94-ae937377708b","Type":"ContainerDied","Data":"a3961aa54255c8969a872cc6c2795415c4ab2a2d6b034c9eb4f2b3f6bbda05e6"}
Oct 30 00:59:43.619363 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.619254    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx" event={"ID":"d42228bf-b9c4-4be5-bc94-ae937377708b","Type":"ContainerDied","Data":"651b07a4ccd0b9bba2f7a415c6e2fea838d572b933f746916da069514a7c5199"}
Oct 30 00:59:43.619363 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.619284    4393 scope.go:122] "RemoveContainer" containerID="a3961aa54255c8969a872cc6c2795415c4ab2a2d6b034c9eb4f2b3f6bbda05e6"
Oct 30 00:59:43.619556 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.619451    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"
Oct 30 00:59:43.630626 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.630541    4393 generic.go:356] "Generic (PLEG): container finished" podID="3b626471-54fd-4301-89fa-cc1738c35cce" containerID="f5ea02f1a8c1d0a0248db155606b648e531de07392eda5cdc00fc4edead745a8" exitCode=0
Oct 30 00:59:43.630715 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.630675    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6" event={"ID":"3b626471-54fd-4301-89fa-cc1738c35cce","Type":"ContainerDied","Data":"f5ea02f1a8c1d0a0248db155606b648e531de07392eda5cdc00fc4edead745a8"}
Oct 30 00:59:43.630930 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.630726    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6" event={"ID":"3b626471-54fd-4301-89fa-cc1738c35cce","Type":"ContainerDied","Data":"340e58de3e26202d8e20f6d1f10a6ae1abf8afc31c7340a47e77ffbae35b3715"}
Oct 30 00:59:43.630930 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.630855    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"
Oct 30 00:59:43.643804 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.643752    4393 scope.go:122] "RemoveContainer" containerID="a3961aa54255c8969a872cc6c2795415c4ab2a2d6b034c9eb4f2b3f6bbda05e6"
Oct 30 00:59:43.644163 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:59:43.644134    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3961aa54255c8969a872cc6c2795415c4ab2a2d6b034c9eb4f2b3f6bbda05e6\": container with ID starting with a3961aa54255c8969a872cc6c2795415c4ab2a2d6b034c9eb4f2b3f6bbda05e6 not found: ID does not exist" containerID="a3961aa54255c8969a872cc6c2795415c4ab2a2d6b034c9eb4f2b3f6bbda05e6"
Oct 30 00:59:43.644266 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.644168    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3961aa54255c8969a872cc6c2795415c4ab2a2d6b034c9eb4f2b3f6bbda05e6"} err="failed to get container status \"a3961aa54255c8969a872cc6c2795415c4ab2a2d6b034c9eb4f2b3f6bbda05e6\": rpc error: code = NotFound desc = could not find container \"a3961aa54255c8969a872cc6c2795415c4ab2a2d6b034c9eb4f2b3f6bbda05e6\": container with ID starting with a3961aa54255c8969a872cc6c2795415c4ab2a2d6b034c9eb4f2b3f6bbda05e6 not found: ID does not exist"
Oct 30 00:59:43.644266 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.644192    4393 scope.go:122] "RemoveContainer" containerID="f5ea02f1a8c1d0a0248db155606b648e531de07392eda5cdc00fc4edead745a8"
Oct 30 00:59:43.664686 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.664658    4393 scope.go:122] "RemoveContainer" containerID="f5ea02f1a8c1d0a0248db155606b648e531de07392eda5cdc00fc4edead745a8"
Oct 30 00:59:43.665126 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:59:43.665099    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5ea02f1a8c1d0a0248db155606b648e531de07392eda5cdc00fc4edead745a8\": container with ID starting with f5ea02f1a8c1d0a0248db155606b648e531de07392eda5cdc00fc4edead745a8 not found: ID does not exist" containerID="f5ea02f1a8c1d0a0248db155606b648e531de07392eda5cdc00fc4edead745a8"
Oct 30 00:59:43.665206 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.665146    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5ea02f1a8c1d0a0248db155606b648e531de07392eda5cdc00fc4edead745a8"} err="failed to get container status \"f5ea02f1a8c1d0a0248db155606b648e531de07392eda5cdc00fc4edead745a8\": rpc error: code = NotFound desc = could not find container \"f5ea02f1a8c1d0a0248db155606b648e531de07392eda5cdc00fc4edead745a8\": container with ID starting with f5ea02f1a8c1d0a0248db155606b648e531de07392eda5cdc00fc4edead745a8 not found: ID does not exist"
Oct 30 00:59:43.674391 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.674366    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b626471-54fd-4301-89fa-cc1738c35cce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b626471-54fd-4301-89fa-cc1738c35cce-istio-data\") pod \"3b626471-54fd-4301-89fa-cc1738c35cce\" (UID: \"3b626471-54fd-4301-89fa-cc1738c35cce\") "
Oct 30 00:59:43.674520 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.674418    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b626471-54fd-4301-89fa-cc1738c35cce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b626471-54fd-4301-89fa-cc1738c35cce-istiod-ca-cert\") pod \"3b626471-54fd-4301-89fa-cc1738c35cce\" (UID: \"3b626471-54fd-4301-89fa-cc1738c35cce\") "
Oct 30 00:59:43.674596 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.674524    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b626471-54fd-4301-89fa-cc1738c35cce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b626471-54fd-4301-89fa-cc1738c35cce-workload-certs\") pod \"3b626471-54fd-4301-89fa-cc1738c35cce\" (UID: \"3b626471-54fd-4301-89fa-cc1738c35cce\") "
Oct 30 00:59:43.674596 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.674557    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b626471-54fd-4301-89fa-cc1738c35cce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b626471-54fd-4301-89fa-cc1738c35cce-workload-socket\") pod \"3b626471-54fd-4301-89fa-cc1738c35cce\" (UID: \"3b626471-54fd-4301-89fa-cc1738c35cce\") "
Oct 30 00:59:43.674719 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.674663    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b626471-54fd-4301-89fa-cc1738c35cce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b626471-54fd-4301-89fa-cc1738c35cce-istio-podinfo\") pod \"3b626471-54fd-4301-89fa-cc1738c35cce\" (UID: \"3b626471-54fd-4301-89fa-cc1738c35cce\") "
Oct 30 00:59:43.674861 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.674841    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b626471-54fd-4301-89fa-cc1738c35cce-istio-token\" (UniqueName: \"kubernetes.io/projected/3b626471-54fd-4301-89fa-cc1738c35cce-istio-token\") pod \"3b626471-54fd-4301-89fa-cc1738c35cce\" (UID: \"3b626471-54fd-4301-89fa-cc1738c35cce\") "
Oct 30 00:59:43.674937 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.674662    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b626471-54fd-4301-89fa-cc1738c35cce-istio-data" pod "3b626471-54fd-4301-89fa-cc1738c35cce" (UID: "3b626471-54fd-4301-89fa-cc1738c35cce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:43.674998 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.674904    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b626471-54fd-4301-89fa-cc1738c35cce-workload-socket" pod "3b626471-54fd-4301-89fa-cc1738c35cce" (UID: "3b626471-54fd-4301-89fa-cc1738c35cce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:43.674998 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.674983    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b626471-54fd-4301-89fa-cc1738c35cce-kube-api-access-jhrzq\" (UniqueName: \"kubernetes.io/projected/3b626471-54fd-4301-89fa-cc1738c35cce-kube-api-access-jhrzq\") pod \"3b626471-54fd-4301-89fa-cc1738c35cce\" (UID: \"3b626471-54fd-4301-89fa-cc1738c35cce\") "
Oct 30 00:59:43.675118 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.675031    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b626471-54fd-4301-89fa-cc1738c35cce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b626471-54fd-4301-89fa-cc1738c35cce-istio-envoy\") pod \"3b626471-54fd-4301-89fa-cc1738c35cce\" (UID: \"3b626471-54fd-4301-89fa-cc1738c35cce\") "
Oct 30 00:59:43.675118 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.675025    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b626471-54fd-4301-89fa-cc1738c35cce-workload-certs" pod "3b626471-54fd-4301-89fa-cc1738c35cce" (UID: "3b626471-54fd-4301-89fa-cc1738c35cce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:43.675118 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.675070    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b626471-54fd-4301-89fa-cc1738c35cce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b626471-54fd-4301-89fa-cc1738c35cce-credential-socket\") pod \"3b626471-54fd-4301-89fa-cc1738c35cce\" (UID: \"3b626471-54fd-4301-89fa-cc1738c35cce\") "
Oct 30 00:59:43.675296 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.675264    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b626471-54fd-4301-89fa-cc1738c35cce-istiod-ca-cert" pod "3b626471-54fd-4301-89fa-cc1738c35cce" (UID: "3b626471-54fd-4301-89fa-cc1738c35cce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 00:59:43.676201 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.676174    4393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b626471-54fd-4301-89fa-cc1738c35cce-workload-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.676307 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.676214    4393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b626471-54fd-4301-89fa-cc1738c35cce-workload-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.676307 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.676230    4393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b626471-54fd-4301-89fa-cc1738c35cce-istio-data\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.676307 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.676245    4393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b626471-54fd-4301-89fa-cc1738c35cce-istiod-ca-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.676467 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.676406    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b626471-54fd-4301-89fa-cc1738c35cce-credential-socket" pod "3b626471-54fd-4301-89fa-cc1738c35cce" (UID: "3b626471-54fd-4301-89fa-cc1738c35cce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:43.678144 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.678035    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b626471-54fd-4301-89fa-cc1738c35cce-istio-token" pod "3b626471-54fd-4301-89fa-cc1738c35cce" (UID: "3b626471-54fd-4301-89fa-cc1738c35cce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:43.678144 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.678101    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b626471-54fd-4301-89fa-cc1738c35cce-istio-podinfo" pod "3b626471-54fd-4301-89fa-cc1738c35cce" (UID: "3b626471-54fd-4301-89fa-cc1738c35cce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 00:59:43.678475 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.678450    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b626471-54fd-4301-89fa-cc1738c35cce-kube-api-access-jhrzq" pod "3b626471-54fd-4301-89fa-cc1738c35cce" (UID: "3b626471-54fd-4301-89fa-cc1738c35cce"). InnerVolumeSpecName "kube-api-access-jhrzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 00:59:43.679523 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.679490    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b626471-54fd-4301-89fa-cc1738c35cce-istio-envoy" pod "3b626471-54fd-4301-89fa-cc1738c35cce" (UID: "3b626471-54fd-4301-89fa-cc1738c35cce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 00:59:43.697670 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.697644    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"]
Oct 30 00:59:43.704394 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.704366    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bmwfx"]
Oct 30 00:59:43.727293 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.727267    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d42228bf-b9c4-4be5-bc94-ae937377708b" path="/var/lib/kubelet/pods/d42228bf-b9c4-4be5-bc94-ae937377708b/volumes"
Oct 30 00:59:43.778032 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.777998    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b626471-54fd-4301-89fa-cc1738c35cce-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.778032 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.778027    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhrzq\" (UniqueName: \"kubernetes.io/projected/3b626471-54fd-4301-89fa-cc1738c35cce-kube-api-access-jhrzq\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.778032 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.778038    4393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b626471-54fd-4301-89fa-cc1738c35cce-istio-envoy\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.778293 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.778048    4393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b626471-54fd-4301-89fa-cc1738c35cce-credential-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.778293 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.778057    4393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b626471-54fd-4301-89fa-cc1738c35cce-istio-podinfo\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 00:59:43.971141 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.971071    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"]
Oct 30 00:59:43.979999 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:43.979968    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-2h2n6"]
Oct 30 00:59:45.726851 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:45.726812    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b626471-54fd-4301-89fa-cc1738c35cce" path="/var/lib/kubelet/pods/3b626471-54fd-4301-89fa-cc1738c35cce/volumes"
Oct 30 00:59:48.470709 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:48.470676    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2"]
Oct 30 00:59:48.472061 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:48.472037    4393 cpu_manager.go:414] "containerMap: removing container" podUID="3b626471-54fd-4301-89fa-cc1738c35cce" containerName="istio-proxy"
Oct 30 00:59:48.472140 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:48.472065    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b626471-54fd-4301-89fa-cc1738c35cce" containerName="istio-proxy"
Oct 30 00:59:48.472140 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:48.472112    4393 cpu_manager.go:414] "containerMap: removing container" podUID="d42228bf-b9c4-4be5-bc94-ae937377708b" containerName="istio-proxy"
Oct 30 00:59:48.472140 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:48.472120    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d42228bf-b9c4-4be5-bc94-ae937377708b" containerName="istio-proxy"
Oct 30 00:59:48.472426 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:48.472407    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b626471-54fd-4301-89fa-cc1738c35cce" containerName="istio-proxy"
Oct 30 00:59:48.472478 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:48.472438    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="d42228bf-b9c4-4be5-bc94-ae937377708b" containerName="istio-proxy"
Oct 30 00:59:48.477527 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:48.477507    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2"
Oct 30 00:59:48.493469 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:48.493439    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2"]
Oct 30 00:59:48.517320 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:48.517296    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-5007e1-serving-cert\""
Oct 30 00:59:48.563497 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:48.563468    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eabc28b8-fab2-42e2-abc8-d69c3fd22959-metrics-certs\") pod \"maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2\" (UID: \"eabc28b8-fab2-42e2-abc8-d69c3fd22959\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2"
Oct 30 00:59:48.563718 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:48.563696    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eabc28b8-fab2-42e2-abc8-d69c3fd22959-maas-api-tls\") pod \"maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2\" (UID: \"eabc28b8-fab2-42e2-abc8-d69c3fd22959\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2"
Oct 30 00:59:48.563946 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:48.563921    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eabc28b8-fab2-42e2-abc8-d69c3fd22959-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2\" (UID: \"eabc28b8-fab2-42e2-abc8-d69c3fd22959\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2"
Oct 30 00:59:48.564006 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:48.563963    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxlg8\" (UniqueName: \"kubernetes.io/projected/eabc28b8-fab2-42e2-abc8-d69c3fd22959-kube-api-access-lxlg8\") pod \"maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2\" (UID: \"eabc28b8-fab2-42e2-abc8-d69c3fd22959\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2"
Oct 30 00:59:48.665205 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:48.665161    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eabc28b8-fab2-42e2-abc8-d69c3fd22959-maas-api-tls\") pod \"maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2\" (UID: \"eabc28b8-fab2-42e2-abc8-d69c3fd22959\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2"
Oct 30 00:59:48.665370 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:48.665295    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eabc28b8-fab2-42e2-abc8-d69c3fd22959-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2\" (UID: \"eabc28b8-fab2-42e2-abc8-d69c3fd22959\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2"
Oct 30 00:59:48.665370 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:48.665316    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxlg8\" (UniqueName: \"kubernetes.io/projected/eabc28b8-fab2-42e2-abc8-d69c3fd22959-kube-api-access-lxlg8\") pod \"maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2\" (UID: \"eabc28b8-fab2-42e2-abc8-d69c3fd22959\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2"
Oct 30 00:59:48.665452 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:48.665369    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eabc28b8-fab2-42e2-abc8-d69c3fd22959-metrics-certs\") pod \"maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2\" (UID: \"eabc28b8-fab2-42e2-abc8-d69c3fd22959\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2"
Oct 30 00:59:48.665978 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:48.665951    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eabc28b8-fab2-42e2-abc8-d69c3fd22959-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2\" (UID: \"eabc28b8-fab2-42e2-abc8-d69c3fd22959\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2"
Oct 30 00:59:48.667794 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:48.667760    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eabc28b8-fab2-42e2-abc8-d69c3fd22959-maas-api-tls\") pod \"maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2\" (UID: \"eabc28b8-fab2-42e2-abc8-d69c3fd22959\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2"
Oct 30 00:59:48.667984 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:48.667964    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eabc28b8-fab2-42e2-abc8-d69c3fd22959-metrics-certs\") pod \"maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2\" (UID: \"eabc28b8-fab2-42e2-abc8-d69c3fd22959\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2"
Oct 30 00:59:48.689407 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:48.689371    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxlg8\" (UniqueName: \"kubernetes.io/projected/eabc28b8-fab2-42e2-abc8-d69c3fd22959-kube-api-access-lxlg8\") pod \"maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2\" (UID: \"eabc28b8-fab2-42e2-abc8-d69c3fd22959\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2"
Oct 30 00:59:48.823805 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:48.823691    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2"
Oct 30 00:59:49.134160 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:49.134130    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2"]
Oct 30 00:59:49.134266 ip-10-0-135-56 kubenswrapper[4393]: W1030 00:59:49.134190    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeabc28b8_fab2_42e2_abc8_d69c3fd22959.slice/crio-ff58159af840065eae9d3d233252fab6d54024ab7f7489e36f66cbe6c71bb8f3 WatchSource:0}: Error finding container ff58159af840065eae9d3d233252fab6d54024ab7f7489e36f66cbe6c71bb8f3: Status 404 returned error can't find the container with id ff58159af840065eae9d3d233252fab6d54024ab7f7489e36f66cbe6c71bb8f3
Oct 30 00:59:49.135693 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:49.135673    4393 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 00:59:49.815104 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:49.815017    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2"
Oct 30 00:59:49.815664 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:49.815638    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2" event={"ID":"eabc28b8-fab2-42e2-abc8-d69c3fd22959","Type":"ContainerStarted","Data":"511f10e569d7fd905c50200b0fd157704c0e663207529993b43f8356407ce895"}
Oct 30 00:59:49.815895 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:49.815870    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2" event={"ID":"eabc28b8-fab2-42e2-abc8-d69c3fd22959","Type":"ContainerStarted","Data":"ff58159af840065eae9d3d233252fab6d54024ab7f7489e36f66cbe6c71bb8f3"}
Oct 30 00:59:49.849258 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:49.849194    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2" podStartSLOduration=1.515351898 podStartE2EDuration="1.849181133s" totalImagesPullingTime="333.829235ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 00:59:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 00:59:49.847832906 +0000 UTC m=+1680.235509294" watchObservedRunningTime="2026-10-30 00:59:49.849181133 +0000 UTC m=+1680.236857522"
Oct 30 00:59:56.554856 ip-10-0-135-56 kubenswrapper[4393]: E1030 00:59:56.554814    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 00:59:56.886002 ip-10-0-135-56 kubenswrapper[4393]: I1030 00:59:56.885968    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2"
Oct 30 01:00:26.513651 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:00:26.513611    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:00:35.873981 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:35.873942    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g"]
Oct 30 01:00:35.874616 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:35.874226    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g" podUID="3385764c-37fe-43ba-afb7-f085f5dfab40" containerName="maas-api" containerID="cri-o://eb158a72650e6f0c36a02a50c655ed185ccd0fcb7498c549e86985ecfabf503e" gracePeriod=30
Oct 30 01:00:36.488634 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:36.488539    4393 generic.go:356] "Generic (PLEG): container finished" podID="3385764c-37fe-43ba-afb7-f085f5dfab40" containerID="eb158a72650e6f0c36a02a50c655ed185ccd0fcb7498c549e86985ecfabf503e" exitCode=0
Oct 30 01:00:36.489319 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:36.489273    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g" event={"ID":"3385764c-37fe-43ba-afb7-f085f5dfab40","Type":"ContainerDied","Data":"eb158a72650e6f0c36a02a50c655ed185ccd0fcb7498c549e86985ecfabf503e"}
Oct 30 01:00:36.489562 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:36.489527    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g" event={"ID":"3385764c-37fe-43ba-afb7-f085f5dfab40","Type":"ContainerDied","Data":"757d93975193e481956d4a6c6575269354451eae0367d1521c88710f9300af3d"}
Oct 30 01:00:36.489739 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:36.489722    4393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="757d93975193e481956d4a6c6575269354451eae0367d1521c88710f9300af3d"
Oct 30 01:00:36.507544 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:36.507507    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g"
Oct 30 01:00:36.661846 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:36.661809    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3385764c-37fe-43ba-afb7-f085f5dfab40-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3385764c-37fe-43ba-afb7-f085f5dfab40-maas-api-tls\") pod \"3385764c-37fe-43ba-afb7-f085f5dfab40\" (UID: \"3385764c-37fe-43ba-afb7-f085f5dfab40\") "
Oct 30 01:00:36.675402 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:36.675365    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3385764c-37fe-43ba-afb7-f085f5dfab40-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3385764c-37fe-43ba-afb7-f085f5dfab40-metrics-certs\") pod \"3385764c-37fe-43ba-afb7-f085f5dfab40\" (UID: \"3385764c-37fe-43ba-afb7-f085f5dfab40\") "
Oct 30 01:00:36.675596 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:36.675443    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3385764c-37fe-43ba-afb7-f085f5dfab40-kube-api-access-dcbfk\" (UniqueName: \"kubernetes.io/projected/3385764c-37fe-43ba-afb7-f085f5dfab40-kube-api-access-dcbfk\") pod \"3385764c-37fe-43ba-afb7-f085f5dfab40\" (UID: \"3385764c-37fe-43ba-afb7-f085f5dfab40\") "
Oct 30 01:00:36.675596 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:36.675571    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3385764c-37fe-43ba-afb7-f085f5dfab40-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3385764c-37fe-43ba-afb7-f085f5dfab40-openshift-service-ca\") pod \"3385764c-37fe-43ba-afb7-f085f5dfab40\" (UID: \"3385764c-37fe-43ba-afb7-f085f5dfab40\") "
Oct 30 01:00:36.676599 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:36.676566    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3385764c-37fe-43ba-afb7-f085f5dfab40-openshift-service-ca" pod "3385764c-37fe-43ba-afb7-f085f5dfab40" (UID: "3385764c-37fe-43ba-afb7-f085f5dfab40"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:00:36.677455 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:36.677375    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3385764c-37fe-43ba-afb7-f085f5dfab40-maas-api-tls" pod "3385764c-37fe-43ba-afb7-f085f5dfab40" (UID: "3385764c-37fe-43ba-afb7-f085f5dfab40"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:00:36.677455 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:36.677382    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3385764c-37fe-43ba-afb7-f085f5dfab40-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:00:36.678335 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:36.678310    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3385764c-37fe-43ba-afb7-f085f5dfab40-metrics-certs" pod "3385764c-37fe-43ba-afb7-f085f5dfab40" (UID: "3385764c-37fe-43ba-afb7-f085f5dfab40"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:00:36.679020 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:36.678994    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3385764c-37fe-43ba-afb7-f085f5dfab40-kube-api-access-dcbfk" pod "3385764c-37fe-43ba-afb7-f085f5dfab40" (UID: "3385764c-37fe-43ba-afb7-f085f5dfab40"). InnerVolumeSpecName "kube-api-access-dcbfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:00:36.778555 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:36.778467    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3385764c-37fe-43ba-afb7-f085f5dfab40-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:00:36.778555 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:36.778498    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3385764c-37fe-43ba-afb7-f085f5dfab40-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:00:36.778555 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:36.778507    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcbfk\" (UniqueName: \"kubernetes.io/projected/3385764c-37fe-43ba-afb7-f085f5dfab40-kube-api-access-dcbfk\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:00:37.498992 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:37.498961    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g"
Oct 30 01:00:37.546983 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:37.546954    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g"]
Oct 30 01:00:37.551436 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:37.551413    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bab19a-58bd9954d5-jrh7g"]
Oct 30 01:00:37.728833 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:37.728766    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3385764c-37fe-43ba-afb7-f085f5dfab40" path="/var/lib/kubelet/pods/3385764c-37fe-43ba-afb7-f085f5dfab40/volumes"
Oct 30 01:00:52.667386 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:52.667346    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls"]
Oct 30 01:00:56.512392 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:00:56.512356    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:00:57.870818 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:57.870763    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6"]
Oct 30 01:00:57.872685 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:57.872649    4393 cpu_manager.go:414] "containerMap: removing container" podUID="3385764c-37fe-43ba-afb7-f085f5dfab40" containerName="maas-api"
Oct 30 01:00:57.872685 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:57.872679    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3385764c-37fe-43ba-afb7-f085f5dfab40" containerName="maas-api"
Oct 30 01:00:57.872973 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:57.872953    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="3385764c-37fe-43ba-afb7-f085f5dfab40" containerName="maas-api"
Oct 30 01:00:57.877843 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:57.877825    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6"
Oct 30 01:00:57.906008 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:57.902342    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6"]
Oct 30 01:00:57.906113 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:57.906082    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b1f6ed\"/\"default-dockercfg-cncnm\""
Oct 30 01:00:57.907994 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:57.907962    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b1f6ed\"/\"openshift-service-ca.crt\""
Oct 30 01:00:57.908199 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:57.908175    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b1f6ed\"/\"test-model-b1f6ed-kserve-self-signed-certs\""
Oct 30 01:00:57.916906 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:57.916882    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b1f6ed\"/\"kube-root-ca.crt\""
Oct 30 01:00:57.964039 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:57.964007    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-tmp-dir\") pod \"test-model-b1f6ed-kserve-5d547ddd7-whgr6\" (UID: \"5985d486-ea0a-4703-9cf9-c2ba1c55f7ba\") " pod="ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6"
Oct 30 01:00:57.964209 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:57.964137    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-tls-certs\") pod \"test-model-b1f6ed-kserve-5d547ddd7-whgr6\" (UID: \"5985d486-ea0a-4703-9cf9-c2ba1c55f7ba\") " pod="ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6"
Oct 30 01:00:57.964209 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:57.964198    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-model-cache\") pod \"test-model-b1f6ed-kserve-5d547ddd7-whgr6\" (UID: \"5985d486-ea0a-4703-9cf9-c2ba1c55f7ba\") " pod="ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6"
Oct 30 01:00:57.964359 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:57.964338    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4md4\" (UniqueName: \"kubernetes.io/projected/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-kube-api-access-g4md4\") pod \"test-model-b1f6ed-kserve-5d547ddd7-whgr6\" (UID: \"5985d486-ea0a-4703-9cf9-c2ba1c55f7ba\") " pod="ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6"
Oct 30 01:00:57.964469 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:57.964452    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-dshm\") pod \"test-model-b1f6ed-kserve-5d547ddd7-whgr6\" (UID: \"5985d486-ea0a-4703-9cf9-c2ba1c55f7ba\") " pod="ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6"
Oct 30 01:00:57.964556 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:57.964544    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-home\") pod \"test-model-b1f6ed-kserve-5d547ddd7-whgr6\" (UID: \"5985d486-ea0a-4703-9cf9-c2ba1c55f7ba\") " pod="ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6"
Oct 30 01:00:58.066378 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:58.066346    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-dshm\") pod \"test-model-b1f6ed-kserve-5d547ddd7-whgr6\" (UID: \"5985d486-ea0a-4703-9cf9-c2ba1c55f7ba\") " pod="ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6"
Oct 30 01:00:58.066555 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:58.066459    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-home\") pod \"test-model-b1f6ed-kserve-5d547ddd7-whgr6\" (UID: \"5985d486-ea0a-4703-9cf9-c2ba1c55f7ba\") " pod="ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6"
Oct 30 01:00:58.066555 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:58.066516    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-tmp-dir\") pod \"test-model-b1f6ed-kserve-5d547ddd7-whgr6\" (UID: \"5985d486-ea0a-4703-9cf9-c2ba1c55f7ba\") " pod="ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6"
Oct 30 01:00:58.066694 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:58.066630    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-tls-certs\") pod \"test-model-b1f6ed-kserve-5d547ddd7-whgr6\" (UID: \"5985d486-ea0a-4703-9cf9-c2ba1c55f7ba\") " pod="ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6"
Oct 30 01:00:58.066747 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:58.066725    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-model-cache\") pod \"test-model-b1f6ed-kserve-5d547ddd7-whgr6\" (UID: \"5985d486-ea0a-4703-9cf9-c2ba1c55f7ba\") " pod="ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6"
Oct 30 01:00:58.066862 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:58.066844    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4md4\" (UniqueName: \"kubernetes.io/projected/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-kube-api-access-g4md4\") pod \"test-model-b1f6ed-kserve-5d547ddd7-whgr6\" (UID: \"5985d486-ea0a-4703-9cf9-c2ba1c55f7ba\") " pod="ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6"
Oct 30 01:00:58.066964 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:58.066874    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-home\") pod \"test-model-b1f6ed-kserve-5d547ddd7-whgr6\" (UID: \"5985d486-ea0a-4703-9cf9-c2ba1c55f7ba\") " pod="ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6"
Oct 30 01:00:58.066964 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:58.066931    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-tmp-dir\") pod \"test-model-b1f6ed-kserve-5d547ddd7-whgr6\" (UID: \"5985d486-ea0a-4703-9cf9-c2ba1c55f7ba\") " pod="ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6"
Oct 30 01:00:58.067124 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:58.067095    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-model-cache\") pod \"test-model-b1f6ed-kserve-5d547ddd7-whgr6\" (UID: \"5985d486-ea0a-4703-9cf9-c2ba1c55f7ba\") " pod="ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6"
Oct 30 01:00:58.069047 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:58.069026    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-dshm\") pod \"test-model-b1f6ed-kserve-5d547ddd7-whgr6\" (UID: \"5985d486-ea0a-4703-9cf9-c2ba1c55f7ba\") " pod="ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6"
Oct 30 01:00:58.069384 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:58.069367    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-tls-certs\") pod \"test-model-b1f6ed-kserve-5d547ddd7-whgr6\" (UID: \"5985d486-ea0a-4703-9cf9-c2ba1c55f7ba\") " pod="ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6"
Oct 30 01:00:58.088209 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:58.088185    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4md4\" (UniqueName: \"kubernetes.io/projected/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-kube-api-access-g4md4\") pod \"test-model-b1f6ed-kserve-5d547ddd7-whgr6\" (UID: \"5985d486-ea0a-4703-9cf9-c2ba1c55f7ba\") " pod="ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6"
Oct 30 01:00:58.230165 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:58.230078    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6"
Oct 30 01:00:58.635609 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:58.635581    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6"]
Oct 30 01:00:58.636344 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:00:58.636310    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5985d486_ea0a_4703_9cf9_c2ba1c55f7ba.slice/crio-e26e1a1b14e70847ab56c5c19982b6becf420da5a757c775641463cdb16b17dd WatchSource:0}: Error finding container e26e1a1b14e70847ab56c5c19982b6becf420da5a757c775641463cdb16b17dd: Status 404 returned error can't find the container with id e26e1a1b14e70847ab56c5c19982b6becf420da5a757c775641463cdb16b17dd
Oct 30 01:00:58.750394 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:58.750360    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6" event={"ID":"5985d486-ea0a-4703-9cf9-c2ba1c55f7ba","Type":"ContainerStarted","Data":"c70681b9ce3da8784e24537f6cccb357583f869f055b4e77551551319bd49518"}
Oct 30 01:00:58.750577 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:58.750407    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6" event={"ID":"5985d486-ea0a-4703-9cf9-c2ba1c55f7ba","Type":"ContainerStarted","Data":"e26e1a1b14e70847ab56c5c19982b6becf420da5a757c775641463cdb16b17dd"}
Oct 30 01:00:58.831260 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:58.831208    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6" podStartSLOduration=1.831195126 podStartE2EDuration="1.831195126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:00:58.828038499 +0000 UTC m=+1749.215714891" watchObservedRunningTime="2026-10-30 01:00:58.831195126 +0000 UTC m=+1749.218871547"
Oct 30 01:00:59.230738 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:00:59.230709    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6"
Oct 30 01:01:00.743471 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:00.743440    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s"]
Oct 30 01:01:00.763029 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:00.748434    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s"
Oct 30 01:01:00.771449 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:00.771419    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s"]
Oct 30 01:01:00.772331 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:00.772307    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8d2a70\"/\"test-model-8d2a70-kserve-self-signed-certs\""
Oct 30 01:01:00.772794 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:00.772751    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8d2a70\"/\"openshift-service-ca.crt\""
Oct 30 01:01:00.772954 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:00.772837    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8d2a70\"/\"default-dockercfg-44j97\""
Oct 30 01:01:00.773650 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:00.773624    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8d2a70\"/\"kube-root-ca.crt\""
Oct 30 01:01:00.834949 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:00.834861    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7jtv\" (UniqueName: \"kubernetes.io/projected/27318ea7-6cda-457e-adc9-33467978bbbc-kube-api-access-w7jtv\") pod \"test-model-8d2a70-kserve-557bb6c9bc-h766s\" (UID: \"27318ea7-6cda-457e-adc9-33467978bbbc\") " pod="ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s"
Oct 30 01:01:00.835115 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:00.834986    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27318ea7-6cda-457e-adc9-33467978bbbc-model-cache\") pod \"test-model-8d2a70-kserve-557bb6c9bc-h766s\" (UID: \"27318ea7-6cda-457e-adc9-33467978bbbc\") " pod="ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s"
Oct 30 01:01:00.835262 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:00.835234    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27318ea7-6cda-457e-adc9-33467978bbbc-home\") pod \"test-model-8d2a70-kserve-557bb6c9bc-h766s\" (UID: \"27318ea7-6cda-457e-adc9-33467978bbbc\") " pod="ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s"
Oct 30 01:01:00.835441 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:00.835422    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27318ea7-6cda-457e-adc9-33467978bbbc-dshm\") pod \"test-model-8d2a70-kserve-557bb6c9bc-h766s\" (UID: \"27318ea7-6cda-457e-adc9-33467978bbbc\") " pod="ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s"
Oct 30 01:01:00.835516 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:00.835484    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27318ea7-6cda-457e-adc9-33467978bbbc-tls-certs\") pod \"test-model-8d2a70-kserve-557bb6c9bc-h766s\" (UID: \"27318ea7-6cda-457e-adc9-33467978bbbc\") " pod="ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s"
Oct 30 01:01:00.835675 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:00.835652    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27318ea7-6cda-457e-adc9-33467978bbbc-tmp-dir\") pod \"test-model-8d2a70-kserve-557bb6c9bc-h766s\" (UID: \"27318ea7-6cda-457e-adc9-33467978bbbc\") " pod="ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s"
Oct 30 01:01:00.937608 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:00.937577    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27318ea7-6cda-457e-adc9-33467978bbbc-tmp-dir\") pod \"test-model-8d2a70-kserve-557bb6c9bc-h766s\" (UID: \"27318ea7-6cda-457e-adc9-33467978bbbc\") " pod="ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s"
Oct 30 01:01:00.937807 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:00.937671    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7jtv\" (UniqueName: \"kubernetes.io/projected/27318ea7-6cda-457e-adc9-33467978bbbc-kube-api-access-w7jtv\") pod \"test-model-8d2a70-kserve-557bb6c9bc-h766s\" (UID: \"27318ea7-6cda-457e-adc9-33467978bbbc\") " pod="ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s"
Oct 30 01:01:00.937807 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:00.937747    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27318ea7-6cda-457e-adc9-33467978bbbc-model-cache\") pod \"test-model-8d2a70-kserve-557bb6c9bc-h766s\" (UID: \"27318ea7-6cda-457e-adc9-33467978bbbc\") " pod="ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s"
Oct 30 01:01:00.938049 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:00.938023    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27318ea7-6cda-457e-adc9-33467978bbbc-tmp-dir\") pod \"test-model-8d2a70-kserve-557bb6c9bc-h766s\" (UID: \"27318ea7-6cda-457e-adc9-33467978bbbc\") " pod="ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s"
Oct 30 01:01:00.938125 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:00.938103    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27318ea7-6cda-457e-adc9-33467978bbbc-home\") pod \"test-model-8d2a70-kserve-557bb6c9bc-h766s\" (UID: \"27318ea7-6cda-457e-adc9-33467978bbbc\") " pod="ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s"
Oct 30 01:01:00.938167 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:00.938146    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27318ea7-6cda-457e-adc9-33467978bbbc-model-cache\") pod \"test-model-8d2a70-kserve-557bb6c9bc-h766s\" (UID: \"27318ea7-6cda-457e-adc9-33467978bbbc\") " pod="ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s"
Oct 30 01:01:00.938210 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:00.938199    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27318ea7-6cda-457e-adc9-33467978bbbc-dshm\") pod \"test-model-8d2a70-kserve-557bb6c9bc-h766s\" (UID: \"27318ea7-6cda-457e-adc9-33467978bbbc\") " pod="ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s"
Oct 30 01:01:00.938248 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:00.938228    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27318ea7-6cda-457e-adc9-33467978bbbc-tls-certs\") pod \"test-model-8d2a70-kserve-557bb6c9bc-h766s\" (UID: \"27318ea7-6cda-457e-adc9-33467978bbbc\") " pod="ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s"
Oct 30 01:01:00.938397 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:00.938377    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27318ea7-6cda-457e-adc9-33467978bbbc-home\") pod \"test-model-8d2a70-kserve-557bb6c9bc-h766s\" (UID: \"27318ea7-6cda-457e-adc9-33467978bbbc\") " pod="ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s"
Oct 30 01:01:00.940423 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:00.940396    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27318ea7-6cda-457e-adc9-33467978bbbc-dshm\") pod \"test-model-8d2a70-kserve-557bb6c9bc-h766s\" (UID: \"27318ea7-6cda-457e-adc9-33467978bbbc\") " pod="ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s"
Oct 30 01:01:00.940844 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:00.940823    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27318ea7-6cda-457e-adc9-33467978bbbc-tls-certs\") pod \"test-model-8d2a70-kserve-557bb6c9bc-h766s\" (UID: \"27318ea7-6cda-457e-adc9-33467978bbbc\") " pod="ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s"
Oct 30 01:01:00.959971 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:00.959944    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7jtv\" (UniqueName: \"kubernetes.io/projected/27318ea7-6cda-457e-adc9-33467978bbbc-kube-api-access-w7jtv\") pod \"test-model-8d2a70-kserve-557bb6c9bc-h766s\" (UID: \"27318ea7-6cda-457e-adc9-33467978bbbc\") " pod="ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s"
Oct 30 01:01:01.115854 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:01.113110    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s"
Oct 30 01:01:01.479341 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:01.479318    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s"]
Oct 30 01:01:01.479819 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:01:01.479770    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27318ea7_6cda_457e_adc9_33467978bbbc.slice/crio-aa443bb6ff04b73905981062e36e5880346d2020c74fc5c9166f36d2cb0ab294 WatchSource:0}: Error finding container aa443bb6ff04b73905981062e36e5880346d2020c74fc5c9166f36d2cb0ab294: Status 404 returned error can't find the container with id aa443bb6ff04b73905981062e36e5880346d2020c74fc5c9166f36d2cb0ab294
Oct 30 01:01:01.792425 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:01.792328    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s" event={"ID":"27318ea7-6cda-457e-adc9-33467978bbbc","Type":"ContainerStarted","Data":"50ee28355f8f93524eb36392ec71062f06ac5a901b4c615e7fe01ceb7aa64176"}
Oct 30 01:01:01.792425 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:01.792371    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s" event={"ID":"27318ea7-6cda-457e-adc9-33467978bbbc","Type":"ContainerStarted","Data":"aa443bb6ff04b73905981062e36e5880346d2020c74fc5c9166f36d2cb0ab294"}
Oct 30 01:01:01.833332 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:01.833276    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s" podStartSLOduration=1.8332609469999999 podStartE2EDuration="1.833260947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:01.829354605 +0000 UTC m=+1752.217030993" watchObservedRunningTime="2026-10-30 01:01:01.833260947 +0000 UTC m=+1752.220937336"
Oct 30 01:01:02.114212 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:02.114121    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s"
Oct 30 01:01:07.691278 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:07.691232    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls" podUID="6362e34f-f675-4d77-b0b8-08176b9c0f4d" containerName="main" containerID="cri-o://30b53c15c1c1a1ee95f0198f39c0c5935b04a8c31b8e9035e0602449ab52d6ff" gracePeriod=45
Oct 30 01:01:07.844662 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:07.844622    4393 generic.go:356] "Generic (PLEG): container finished" podID="6362e34f-f675-4d77-b0b8-08176b9c0f4d" containerID="30b53c15c1c1a1ee95f0198f39c0c5935b04a8c31b8e9035e0602449ab52d6ff" exitCode=0
Oct 30 01:01:07.844848 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:07.844689    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls" event={"ID":"6362e34f-f675-4d77-b0b8-08176b9c0f4d","Type":"ContainerDied","Data":"30b53c15c1c1a1ee95f0198f39c0c5935b04a8c31b8e9035e0602449ab52d6ff"}
Oct 30 01:01:08.023404 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:08.023375    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls"
Oct 30 01:01:08.125638 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:08.125604    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6362e34f-f675-4d77-b0b8-08176b9c0f4d-home\" (UniqueName: \"kubernetes.io/empty-dir/6362e34f-f675-4d77-b0b8-08176b9c0f4d-home\") pod \"6362e34f-f675-4d77-b0b8-08176b9c0f4d\" (UID: \"6362e34f-f675-4d77-b0b8-08176b9c0f4d\") "
Oct 30 01:01:08.125984 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:08.125744    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6362e34f-f675-4d77-b0b8-08176b9c0f4d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6362e34f-f675-4d77-b0b8-08176b9c0f4d-dshm\") pod \"6362e34f-f675-4d77-b0b8-08176b9c0f4d\" (UID: \"6362e34f-f675-4d77-b0b8-08176b9c0f4d\") "
Oct 30 01:01:08.125984 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:08.125932    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6362e34f-f675-4d77-b0b8-08176b9c0f4d-home" pod "6362e34f-f675-4d77-b0b8-08176b9c0f4d" (UID: "6362e34f-f675-4d77-b0b8-08176b9c0f4d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:08.125984 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:08.125938    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6362e34f-f675-4d77-b0b8-08176b9c0f4d-kube-api-access-h4wjr\" (UniqueName: \"kubernetes.io/projected/6362e34f-f675-4d77-b0b8-08176b9c0f4d-kube-api-access-h4wjr\") pod \"6362e34f-f675-4d77-b0b8-08176b9c0f4d\" (UID: \"6362e34f-f675-4d77-b0b8-08176b9c0f4d\") "
Oct 30 01:01:08.126214 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:08.126026    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6362e34f-f675-4d77-b0b8-08176b9c0f4d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6362e34f-f675-4d77-b0b8-08176b9c0f4d-tmp-dir\") pod \"6362e34f-f675-4d77-b0b8-08176b9c0f4d\" (UID: \"6362e34f-f675-4d77-b0b8-08176b9c0f4d\") "
Oct 30 01:01:08.126214 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:08.126139    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6362e34f-f675-4d77-b0b8-08176b9c0f4d-tls-certs\" (UniqueName: \"kubernetes.io/secret/6362e34f-f675-4d77-b0b8-08176b9c0f4d-tls-certs\") pod \"6362e34f-f675-4d77-b0b8-08176b9c0f4d\" (UID: \"6362e34f-f675-4d77-b0b8-08176b9c0f4d\") "
Oct 30 01:01:08.126325 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:08.126284    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6362e34f-f675-4d77-b0b8-08176b9c0f4d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6362e34f-f675-4d77-b0b8-08176b9c0f4d-model-cache\") pod \"6362e34f-f675-4d77-b0b8-08176b9c0f4d\" (UID: \"6362e34f-f675-4d77-b0b8-08176b9c0f4d\") "
Oct 30 01:01:08.127382 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:08.127137    4393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6362e34f-f675-4d77-b0b8-08176b9c0f4d-home\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:01:08.127499 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:08.127403    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6362e34f-f675-4d77-b0b8-08176b9c0f4d-model-cache" pod "6362e34f-f675-4d77-b0b8-08176b9c0f4d" (UID: "6362e34f-f675-4d77-b0b8-08176b9c0f4d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:08.127675 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:08.127622    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6362e34f-f675-4d77-b0b8-08176b9c0f4d-tmp-dir" pod "6362e34f-f675-4d77-b0b8-08176b9c0f4d" (UID: "6362e34f-f675-4d77-b0b8-08176b9c0f4d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:08.129834 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:08.129806    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6362e34f-f675-4d77-b0b8-08176b9c0f4d-kube-api-access-h4wjr" pod "6362e34f-f675-4d77-b0b8-08176b9c0f4d" (UID: "6362e34f-f675-4d77-b0b8-08176b9c0f4d"). InnerVolumeSpecName "kube-api-access-h4wjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:01:08.130284 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:08.130259    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6362e34f-f675-4d77-b0b8-08176b9c0f4d-tls-certs" pod "6362e34f-f675-4d77-b0b8-08176b9c0f4d" (UID: "6362e34f-f675-4d77-b0b8-08176b9c0f4d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:01:08.130284 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:08.130276    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6362e34f-f675-4d77-b0b8-08176b9c0f4d-dshm" pod "6362e34f-f675-4d77-b0b8-08176b9c0f4d" (UID: "6362e34f-f675-4d77-b0b8-08176b9c0f4d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:08.228448 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:08.228366    4393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6362e34f-f675-4d77-b0b8-08176b9c0f4d-tls-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:01:08.228448 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:08.228399    4393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6362e34f-f675-4d77-b0b8-08176b9c0f4d-model-cache\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:01:08.228448 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:08.228413    4393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6362e34f-f675-4d77-b0b8-08176b9c0f4d-dshm\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:01:08.228448 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:08.228422    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4wjr\" (UniqueName: \"kubernetes.io/projected/6362e34f-f675-4d77-b0b8-08176b9c0f4d-kube-api-access-h4wjr\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:01:08.228684 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:08.228460    4393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6362e34f-f675-4d77-b0b8-08176b9c0f4d-tmp-dir\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:01:08.232153 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:08.232132    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6"
Oct 30 01:01:08.245595 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:08.245553    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6"
Oct 30 01:01:08.854723 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:08.854693    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls"
Oct 30 01:01:08.855183 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:08.854737    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls" event={"ID":"6362e34f-f675-4d77-b0b8-08176b9c0f4d","Type":"ContainerDied","Data":"4a9c41804d30dbbb842f206a381175d06a7e3ad2e8b75f9bf641c1a9fdbdbff7"}
Oct 30 01:01:08.855183 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:08.854803    4393 scope.go:122] "RemoveContainer" containerID="30b53c15c1c1a1ee95f0198f39c0c5935b04a8c31b8e9035e0602449ab52d6ff"
Oct 30 01:01:08.871862 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:08.871459    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6"
Oct 30 01:01:08.905821 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:08.905796    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls"]
Oct 30 01:01:08.911090 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:08.911061    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-6444d7/test-model-6444d7-kserve-798d574689-xpmls"]
Oct 30 01:01:09.731428 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:09.731388    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6362e34f-f675-4d77-b0b8-08176b9c0f4d" path="/var/lib/kubelet/pods/6362e34f-f675-4d77-b0b8-08176b9c0f4d/volumes"
Oct 30 01:01:11.114192 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:11.114158    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s"
Oct 30 01:01:11.128110 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:11.128082    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s"
Oct 30 01:01:11.895148 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:11.895107    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s"
Oct 30 01:01:17.181601 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:17.177634    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf"]
Oct 30 01:01:18.610720 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:18.609890    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb"]
Oct 30 01:01:18.611291 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:18.611268    4393 cpu_manager.go:414] "containerMap: removing container" podUID="6362e34f-f675-4d77-b0b8-08176b9c0f4d" containerName="main"
Oct 30 01:01:18.611377 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:18.611292    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6362e34f-f675-4d77-b0b8-08176b9c0f4d" containerName="main"
Oct 30 01:01:18.611497 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:18.611482    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="6362e34f-f675-4d77-b0b8-08176b9c0f4d" containerName="main"
Oct 30 01:01:18.615719 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:18.615688    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb"
Oct 30 01:01:18.660675 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:18.660639    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb"]
Oct 30 01:01:18.664828 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:18.664806    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5007e1\"/\"openshift-service-ca.crt\""
Oct 30 01:01:18.665016 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:18.664986    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5007e1\"/\"test-model-5007e1-kserve-self-signed-certs\""
Oct 30 01:01:18.665016 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:18.664996    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5007e1\"/\"kube-root-ca.crt\""
Oct 30 01:01:18.665214 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:18.665050    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5007e1\"/\"default-dockercfg-zrzpc\""
Oct 30 01:01:18.746045 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:18.746002    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b136c12-ff27-4e1a-8f06-786770f0cb4a-model-cache\") pod \"test-model-5007e1-kserve-6d4b6f4955-hw4jb\" (UID: \"0b136c12-ff27-4e1a-8f06-786770f0cb4a\") " pod="ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb"
Oct 30 01:01:18.746250 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:18.746101    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b136c12-ff27-4e1a-8f06-786770f0cb4a-tmp-dir\") pod \"test-model-5007e1-kserve-6d4b6f4955-hw4jb\" (UID: \"0b136c12-ff27-4e1a-8f06-786770f0cb4a\") " pod="ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb"
Oct 30 01:01:18.746250 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:18.746210    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b136c12-ff27-4e1a-8f06-786770f0cb4a-home\") pod \"test-model-5007e1-kserve-6d4b6f4955-hw4jb\" (UID: \"0b136c12-ff27-4e1a-8f06-786770f0cb4a\") " pod="ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb"
Oct 30 01:01:18.746250 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:18.746243    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbb9z\" (UniqueName: \"kubernetes.io/projected/0b136c12-ff27-4e1a-8f06-786770f0cb4a-kube-api-access-vbb9z\") pod \"test-model-5007e1-kserve-6d4b6f4955-hw4jb\" (UID: \"0b136c12-ff27-4e1a-8f06-786770f0cb4a\") " pod="ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb"
Oct 30 01:01:18.746473 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:18.746403    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b136c12-ff27-4e1a-8f06-786770f0cb4a-tls-certs\") pod \"test-model-5007e1-kserve-6d4b6f4955-hw4jb\" (UID: \"0b136c12-ff27-4e1a-8f06-786770f0cb4a\") " pod="ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb"
Oct 30 01:01:18.746473 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:18.746442    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b136c12-ff27-4e1a-8f06-786770f0cb4a-dshm\") pod \"test-model-5007e1-kserve-6d4b6f4955-hw4jb\" (UID: \"0b136c12-ff27-4e1a-8f06-786770f0cb4a\") " pod="ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb"
Oct 30 01:01:18.870868 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:18.859094    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b136c12-ff27-4e1a-8f06-786770f0cb4a-home\") pod \"test-model-5007e1-kserve-6d4b6f4955-hw4jb\" (UID: \"0b136c12-ff27-4e1a-8f06-786770f0cb4a\") " pod="ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb"
Oct 30 01:01:18.870868 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:18.859146    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbb9z\" (UniqueName: \"kubernetes.io/projected/0b136c12-ff27-4e1a-8f06-786770f0cb4a-kube-api-access-vbb9z\") pod \"test-model-5007e1-kserve-6d4b6f4955-hw4jb\" (UID: \"0b136c12-ff27-4e1a-8f06-786770f0cb4a\") " pod="ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb"
Oct 30 01:01:18.870868 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:18.859413    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b136c12-ff27-4e1a-8f06-786770f0cb4a-tls-certs\") pod \"test-model-5007e1-kserve-6d4b6f4955-hw4jb\" (UID: \"0b136c12-ff27-4e1a-8f06-786770f0cb4a\") " pod="ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb"
Oct 30 01:01:18.870868 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:18.859464    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b136c12-ff27-4e1a-8f06-786770f0cb4a-dshm\") pod \"test-model-5007e1-kserve-6d4b6f4955-hw4jb\" (UID: \"0b136c12-ff27-4e1a-8f06-786770f0cb4a\") " pod="ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb"
Oct 30 01:01:18.870868 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:18.859669    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b136c12-ff27-4e1a-8f06-786770f0cb4a-model-cache\") pod \"test-model-5007e1-kserve-6d4b6f4955-hw4jb\" (UID: \"0b136c12-ff27-4e1a-8f06-786770f0cb4a\") " pod="ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb"
Oct 30 01:01:18.870868 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:18.859751    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b136c12-ff27-4e1a-8f06-786770f0cb4a-tmp-dir\") pod \"test-model-5007e1-kserve-6d4b6f4955-hw4jb\" (UID: \"0b136c12-ff27-4e1a-8f06-786770f0cb4a\") " pod="ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb"
Oct 30 01:01:18.894802 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:18.891199    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b136c12-ff27-4e1a-8f06-786770f0cb4a-home\") pod \"test-model-5007e1-kserve-6d4b6f4955-hw4jb\" (UID: \"0b136c12-ff27-4e1a-8f06-786770f0cb4a\") " pod="ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb"
Oct 30 01:01:18.901808 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:18.897376    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b136c12-ff27-4e1a-8f06-786770f0cb4a-tmp-dir\") pod \"test-model-5007e1-kserve-6d4b6f4955-hw4jb\" (UID: \"0b136c12-ff27-4e1a-8f06-786770f0cb4a\") " pod="ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb"
Oct 30 01:01:18.901808 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:18.899135    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b136c12-ff27-4e1a-8f06-786770f0cb4a-model-cache\") pod \"test-model-5007e1-kserve-6d4b6f4955-hw4jb\" (UID: \"0b136c12-ff27-4e1a-8f06-786770f0cb4a\") " pod="ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb"
Oct 30 01:01:18.919807 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:18.916075    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b136c12-ff27-4e1a-8f06-786770f0cb4a-dshm\") pod \"test-model-5007e1-kserve-6d4b6f4955-hw4jb\" (UID: \"0b136c12-ff27-4e1a-8f06-786770f0cb4a\") " pod="ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb"
Oct 30 01:01:18.944807 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:18.926695    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b136c12-ff27-4e1a-8f06-786770f0cb4a-tls-certs\") pod \"test-model-5007e1-kserve-6d4b6f4955-hw4jb\" (UID: \"0b136c12-ff27-4e1a-8f06-786770f0cb4a\") " pod="ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb"
Oct 30 01:01:18.945851 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:18.945705    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbb9z\" (UniqueName: \"kubernetes.io/projected/0b136c12-ff27-4e1a-8f06-786770f0cb4a-kube-api-access-vbb9z\") pod \"test-model-5007e1-kserve-6d4b6f4955-hw4jb\" (UID: \"0b136c12-ff27-4e1a-8f06-786770f0cb4a\") " pod="ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb"
Oct 30 01:01:18.970800 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:18.967149    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb"
Oct 30 01:01:19.854515 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:19.854482    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb"]
Oct 30 01:01:20.093726 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:20.093643    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb" event={"ID":"0b136c12-ff27-4e1a-8f06-786770f0cb4a","Type":"ContainerStarted","Data":"025c293fdb1e6aa5ef164785dce656a16fea3045b0974070d55c4253f4d862a9"}
Oct 30 01:01:20.093726 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:20.093694    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb" event={"ID":"0b136c12-ff27-4e1a-8f06-786770f0cb4a","Type":"ContainerStarted","Data":"8c23d2cbaf1ca1cbaabb8cca39fbbf974b9292e359f9031b686c4c531d61f79a"}
Oct 30 01:01:20.130907 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:20.130862    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb" podStartSLOduration=2.130848271 podStartE2EDuration="2.130848271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:20.130497773 +0000 UTC m=+1770.518174162" watchObservedRunningTime="2026-10-30 01:01:20.130848271 +0000 UTC m=+1770.518524659"
Oct 30 01:01:20.968165 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:20.968131    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb"
Oct 30 01:01:24.025176 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:24.025143    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s"]
Oct 30 01:01:24.031021 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:24.030992    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s"
Oct 30 01:01:24.054761 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:24.053633    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-cce513\"/\"openshift-service-ca.crt\""
Oct 30 01:01:24.054761 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:24.053824    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-cce513\"/\"kube-root-ca.crt\""
Oct 30 01:01:24.054761 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:24.054011    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-cce513\"/\"auth-test-model-cce513-kserve-self-signed-certs\""
Oct 30 01:01:24.054761 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:24.054715    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s"]
Oct 30 01:01:24.057393 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:24.057239    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-cce513\"/\"default-dockercfg-pklnp\""
Oct 30 01:01:24.203192 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:24.203160    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01412f93-3e85-41e2-aa25-fb8be317b84b-home\") pod \"auth-test-model-cce513-kserve-796cd476f7-x9b8s\" (UID: \"01412f93-3e85-41e2-aa25-fb8be317b84b\") " pod="ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s"
Oct 30 01:01:24.203377 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:24.203207    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01412f93-3e85-41e2-aa25-fb8be317b84b-model-cache\") pod \"auth-test-model-cce513-kserve-796cd476f7-x9b8s\" (UID: \"01412f93-3e85-41e2-aa25-fb8be317b84b\") " pod="ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s"
Oct 30 01:01:24.203377 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:24.203344    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01412f93-3e85-41e2-aa25-fb8be317b84b-tls-certs\") pod \"auth-test-model-cce513-kserve-796cd476f7-x9b8s\" (UID: \"01412f93-3e85-41e2-aa25-fb8be317b84b\") " pod="ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s"
Oct 30 01:01:24.203497 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:24.203402    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01412f93-3e85-41e2-aa25-fb8be317b84b-dshm\") pod \"auth-test-model-cce513-kserve-796cd476f7-x9b8s\" (UID: \"01412f93-3e85-41e2-aa25-fb8be317b84b\") " pod="ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s"
Oct 30 01:01:24.203589 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:24.203568    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01412f93-3e85-41e2-aa25-fb8be317b84b-tmp-dir\") pod \"auth-test-model-cce513-kserve-796cd476f7-x9b8s\" (UID: \"01412f93-3e85-41e2-aa25-fb8be317b84b\") " pod="ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s"
Oct 30 01:01:24.203664 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:24.203646    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stjjl\" (UniqueName: \"kubernetes.io/projected/01412f93-3e85-41e2-aa25-fb8be317b84b-kube-api-access-stjjl\") pod \"auth-test-model-cce513-kserve-796cd476f7-x9b8s\" (UID: \"01412f93-3e85-41e2-aa25-fb8be317b84b\") " pod="ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s"
Oct 30 01:01:24.305709 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:24.305630    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01412f93-3e85-41e2-aa25-fb8be317b84b-dshm\") pod \"auth-test-model-cce513-kserve-796cd476f7-x9b8s\" (UID: \"01412f93-3e85-41e2-aa25-fb8be317b84b\") " pod="ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s"
Oct 30 01:01:24.305879 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:24.305734    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01412f93-3e85-41e2-aa25-fb8be317b84b-tmp-dir\") pod \"auth-test-model-cce513-kserve-796cd476f7-x9b8s\" (UID: \"01412f93-3e85-41e2-aa25-fb8be317b84b\") " pod="ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s"
Oct 30 01:01:24.305879 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:24.305814    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stjjl\" (UniqueName: \"kubernetes.io/projected/01412f93-3e85-41e2-aa25-fb8be317b84b-kube-api-access-stjjl\") pod \"auth-test-model-cce513-kserve-796cd476f7-x9b8s\" (UID: \"01412f93-3e85-41e2-aa25-fb8be317b84b\") " pod="ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s"
Oct 30 01:01:24.306059 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:24.306030    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01412f93-3e85-41e2-aa25-fb8be317b84b-home\") pod \"auth-test-model-cce513-kserve-796cd476f7-x9b8s\" (UID: \"01412f93-3e85-41e2-aa25-fb8be317b84b\") " pod="ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s"
Oct 30 01:01:24.306127 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:24.306083    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01412f93-3e85-41e2-aa25-fb8be317b84b-model-cache\") pod \"auth-test-model-cce513-kserve-796cd476f7-x9b8s\" (UID: \"01412f93-3e85-41e2-aa25-fb8be317b84b\") " pod="ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s"
Oct 30 01:01:24.306191 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:24.306161    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01412f93-3e85-41e2-aa25-fb8be317b84b-tmp-dir\") pod \"auth-test-model-cce513-kserve-796cd476f7-x9b8s\" (UID: \"01412f93-3e85-41e2-aa25-fb8be317b84b\") " pod="ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s"
Oct 30 01:01:24.306191 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:24.306169    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01412f93-3e85-41e2-aa25-fb8be317b84b-tls-certs\") pod \"auth-test-model-cce513-kserve-796cd476f7-x9b8s\" (UID: \"01412f93-3e85-41e2-aa25-fb8be317b84b\") " pod="ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s"
Oct 30 01:01:24.306413 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:24.306389    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01412f93-3e85-41e2-aa25-fb8be317b84b-home\") pod \"auth-test-model-cce513-kserve-796cd476f7-x9b8s\" (UID: \"01412f93-3e85-41e2-aa25-fb8be317b84b\") " pod="ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s"
Oct 30 01:01:24.306477 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:24.306465    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01412f93-3e85-41e2-aa25-fb8be317b84b-model-cache\") pod \"auth-test-model-cce513-kserve-796cd476f7-x9b8s\" (UID: \"01412f93-3e85-41e2-aa25-fb8be317b84b\") " pod="ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s"
Oct 30 01:01:24.308214 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:24.308185    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01412f93-3e85-41e2-aa25-fb8be317b84b-dshm\") pod \"auth-test-model-cce513-kserve-796cd476f7-x9b8s\" (UID: \"01412f93-3e85-41e2-aa25-fb8be317b84b\") " pod="ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s"
Oct 30 01:01:24.308665 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:24.308643    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01412f93-3e85-41e2-aa25-fb8be317b84b-tls-certs\") pod \"auth-test-model-cce513-kserve-796cd476f7-x9b8s\" (UID: \"01412f93-3e85-41e2-aa25-fb8be317b84b\") " pod="ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s"
Oct 30 01:01:24.326874 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:24.326853    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stjjl\" (UniqueName: \"kubernetes.io/projected/01412f93-3e85-41e2-aa25-fb8be317b84b-kube-api-access-stjjl\") pod \"auth-test-model-cce513-kserve-796cd476f7-x9b8s\" (UID: \"01412f93-3e85-41e2-aa25-fb8be317b84b\") " pod="ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s"
Oct 30 01:01:24.384181 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:24.384146    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s"
Oct 30 01:01:24.850355 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:24.850303    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s"]
Oct 30 01:01:24.851018 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:01:24.850991    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01412f93_3e85_41e2_aa25_fb8be317b84b.slice/crio-2a1da990a78fb3aecff35ecea05c47c019317ea00810dd0eaae317900238785a WatchSource:0}: Error finding container 2a1da990a78fb3aecff35ecea05c47c019317ea00810dd0eaae317900238785a: Status 404 returned error can't find the container with id 2a1da990a78fb3aecff35ecea05c47c019317ea00810dd0eaae317900238785a
Oct 30 01:01:25.135839 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:25.135755    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s" event={"ID":"01412f93-3e85-41e2-aa25-fb8be317b84b","Type":"ContainerStarted","Data":"514475b867f8488b96281093e5d3890dcb65504ac3cdf41d4a4a800ac1d3629f"}
Oct 30 01:01:25.136334 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:25.135858    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s" event={"ID":"01412f93-3e85-41e2-aa25-fb8be317b84b","Type":"ContainerStarted","Data":"2a1da990a78fb3aecff35ecea05c47c019317ea00810dd0eaae317900238785a"}
Oct 30 01:01:25.176473 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:25.176426    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s" podStartSLOduration=2.176412219 podStartE2EDuration="2.176412219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:01:25.172685407 +0000 UTC m=+1775.560361798" watchObservedRunningTime="2026-10-30 01:01:25.176412219 +0000 UTC m=+1775.564088609"
Oct 30 01:01:25.386253 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:25.386170    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s"
Oct 30 01:01:26.514196 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:01:26.514148    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:01:28.968323 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:28.968291    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb"
Oct 30 01:01:28.980724 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:28.980698    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb"
Oct 30 01:01:29.206979 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:29.206945    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb"
Oct 30 01:01:32.260339 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:32.260298    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf" podUID="62a0449f-5a15-4e71-83aa-49d886d97a79" containerName="main" containerID="cri-o://8263a7846869e6db84446fcca183d0294e5c6906fd9bb82a5469619ed1a87097" gracePeriod=45
Oct 30 01:01:32.557364 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:32.557338    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf"
Oct 30 01:01:32.661954 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:32.661914    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62a0449f-5a15-4e71-83aa-49d886d97a79-dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a0449f-5a15-4e71-83aa-49d886d97a79-dshm\") pod \"62a0449f-5a15-4e71-83aa-49d886d97a79\" (UID: \"62a0449f-5a15-4e71-83aa-49d886d97a79\") "
Oct 30 01:01:32.662222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:32.662207    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62a0449f-5a15-4e71-83aa-49d886d97a79-tls-certs\" (UniqueName: \"kubernetes.io/secret/62a0449f-5a15-4e71-83aa-49d886d97a79-tls-certs\") pod \"62a0449f-5a15-4e71-83aa-49d886d97a79\" (UID: \"62a0449f-5a15-4e71-83aa-49d886d97a79\") "
Oct 30 01:01:32.662501 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:32.662486    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62a0449f-5a15-4e71-83aa-49d886d97a79-kube-api-access-988kk\" (UniqueName: \"kubernetes.io/projected/62a0449f-5a15-4e71-83aa-49d886d97a79-kube-api-access-988kk\") pod \"62a0449f-5a15-4e71-83aa-49d886d97a79\" (UID: \"62a0449f-5a15-4e71-83aa-49d886d97a79\") "
Oct 30 01:01:32.662629 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:32.662614    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62a0449f-5a15-4e71-83aa-49d886d97a79-home\" (UniqueName: \"kubernetes.io/empty-dir/62a0449f-5a15-4e71-83aa-49d886d97a79-home\") pod \"62a0449f-5a15-4e71-83aa-49d886d97a79\" (UID: \"62a0449f-5a15-4e71-83aa-49d886d97a79\") "
Oct 30 01:01:32.662799 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:32.662756    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62a0449f-5a15-4e71-83aa-49d886d97a79-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a0449f-5a15-4e71-83aa-49d886d97a79-model-cache\") pod \"62a0449f-5a15-4e71-83aa-49d886d97a79\" (UID: \"62a0449f-5a15-4e71-83aa-49d886d97a79\") "
Oct 30 01:01:32.662981 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:32.662968    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62a0449f-5a15-4e71-83aa-49d886d97a79-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a0449f-5a15-4e71-83aa-49d886d97a79-tmp-dir\") pod \"62a0449f-5a15-4e71-83aa-49d886d97a79\" (UID: \"62a0449f-5a15-4e71-83aa-49d886d97a79\") "
Oct 30 01:01:32.664302 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:32.664209    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a0449f-5a15-4e71-83aa-49d886d97a79-tmp-dir" pod "62a0449f-5a15-4e71-83aa-49d886d97a79" (UID: "62a0449f-5a15-4e71-83aa-49d886d97a79"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:32.667338 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:32.666668    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a0449f-5a15-4e71-83aa-49d886d97a79-model-cache" pod "62a0449f-5a15-4e71-83aa-49d886d97a79" (UID: "62a0449f-5a15-4e71-83aa-49d886d97a79"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:32.667338 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:32.666731    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a0449f-5a15-4e71-83aa-49d886d97a79-home" pod "62a0449f-5a15-4e71-83aa-49d886d97a79" (UID: "62a0449f-5a15-4e71-83aa-49d886d97a79"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:32.749857 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:32.749818    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a0449f-5a15-4e71-83aa-49d886d97a79-dshm" pod "62a0449f-5a15-4e71-83aa-49d886d97a79" (UID: "62a0449f-5a15-4e71-83aa-49d886d97a79"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:01:32.750071 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:32.749969    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a0449f-5a15-4e71-83aa-49d886d97a79-kube-api-access-988kk" pod "62a0449f-5a15-4e71-83aa-49d886d97a79" (UID: "62a0449f-5a15-4e71-83aa-49d886d97a79"). InnerVolumeSpecName "kube-api-access-988kk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:01:32.753148 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:32.753109    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62a0449f-5a15-4e71-83aa-49d886d97a79-tls-certs" pod "62a0449f-5a15-4e71-83aa-49d886d97a79" (UID: "62a0449f-5a15-4e71-83aa-49d886d97a79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:01:32.765886 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:32.765839    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-988kk\" (UniqueName: \"kubernetes.io/projected/62a0449f-5a15-4e71-83aa-49d886d97a79-kube-api-access-988kk\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:01:32.765886 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:32.765879    4393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a0449f-5a15-4e71-83aa-49d886d97a79-home\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:01:32.766035 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:32.765895    4393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a0449f-5a15-4e71-83aa-49d886d97a79-model-cache\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:01:32.766035 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:32.765909    4393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a0449f-5a15-4e71-83aa-49d886d97a79-tmp-dir\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:01:32.766035 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:32.765920    4393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a0449f-5a15-4e71-83aa-49d886d97a79-dshm\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:01:32.766035 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:32.765931    4393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a0449f-5a15-4e71-83aa-49d886d97a79-tls-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:01:33.230615 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:33.230584    4393 generic.go:356] "Generic (PLEG): container finished" podID="62a0449f-5a15-4e71-83aa-49d886d97a79" containerID="8263a7846869e6db84446fcca183d0294e5c6906fd9bb82a5469619ed1a87097" exitCode=0
Oct 30 01:01:33.230863 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:33.230644    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf"
Oct 30 01:01:33.230863 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:33.230665    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf" event={"ID":"62a0449f-5a15-4e71-83aa-49d886d97a79","Type":"ContainerDied","Data":"8263a7846869e6db84446fcca183d0294e5c6906fd9bb82a5469619ed1a87097"}
Oct 30 01:01:33.230863 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:33.230695    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf" event={"ID":"62a0449f-5a15-4e71-83aa-49d886d97a79","Type":"ContainerDied","Data":"242e6f0e09e993ce87b7cbf233190fb16d9f91c470a5d3419458493616a3719d"}
Oct 30 01:01:33.230863 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:33.230711    4393 scope.go:122] "RemoveContainer" containerID="8263a7846869e6db84446fcca183d0294e5c6906fd9bb82a5469619ed1a87097"
Oct 30 01:01:33.244673 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:33.244651    4393 scope.go:122] "RemoveContainer" containerID="8263a7846869e6db84446fcca183d0294e5c6906fd9bb82a5469619ed1a87097"
Oct 30 01:01:33.245004 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:01:33.244982    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8263a7846869e6db84446fcca183d0294e5c6906fd9bb82a5469619ed1a87097\": container with ID starting with 8263a7846869e6db84446fcca183d0294e5c6906fd9bb82a5469619ed1a87097 not found: ID does not exist" containerID="8263a7846869e6db84446fcca183d0294e5c6906fd9bb82a5469619ed1a87097"
Oct 30 01:01:33.245064 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:33.245014    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8263a7846869e6db84446fcca183d0294e5c6906fd9bb82a5469619ed1a87097"} err="failed to get container status \"8263a7846869e6db84446fcca183d0294e5c6906fd9bb82a5469619ed1a87097\": rpc error: code = NotFound desc = could not find container \"8263a7846869e6db84446fcca183d0294e5c6906fd9bb82a5469619ed1a87097\": container with ID starting with 8263a7846869e6db84446fcca183d0294e5c6906fd9bb82a5469619ed1a87097 not found: ID does not exist"
Oct 30 01:01:33.279194 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:33.279171    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf"]
Oct 30 01:01:33.289043 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:33.289016    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-b3bdf9/test-model-b3bdf9-kserve-776bf69f7f-xgstf"]
Oct 30 01:01:33.727969 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:33.727938    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62a0449f-5a15-4e71-83aa-49d886d97a79" path="/var/lib/kubelet/pods/62a0449f-5a15-4e71-83aa-49d886d97a79/volumes"
Oct 30 01:01:34.385233 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:34.385164    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s"
Oct 30 01:01:34.418819 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:34.418223    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s"
Oct 30 01:01:35.270555 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:01:35.270521    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s"
Oct 30 01:01:56.642716 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:01:56.642671    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:02:07.818194 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:07.818153    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"]
Oct 30 01:02:07.818707 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:07.818408    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h" podUID="141a2288-b7f5-46e6-85cb-c77e432999d3" containerName="istio-proxy" containerID="cri-o://10249610d99c7036916c02e0ffff104f677ffbf06619e8b413c1223498bcd08f" gracePeriod=30
Oct 30 01:02:12.334399 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:12.334361    4393 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:02:12.334861 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:12.334414    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h" podUID="141a2288-b7f5-46e6-85cb-c77e432999d3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:02:13.228709 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.228683    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"
Oct 30 01:02:13.319643 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.319609    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/141a2288-b7f5-46e6-85cb-c77e432999d3-kube-api-access-xmtdn\" (UniqueName: \"kubernetes.io/projected/141a2288-b7f5-46e6-85cb-c77e432999d3-kube-api-access-xmtdn\") pod \"141a2288-b7f5-46e6-85cb-c77e432999d3\" (UID: \"141a2288-b7f5-46e6-85cb-c77e432999d3\") "
Oct 30 01:02:13.319886 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.319829    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/141a2288-b7f5-46e6-85cb-c77e432999d3-istio-token\" (UniqueName: \"kubernetes.io/projected/141a2288-b7f5-46e6-85cb-c77e432999d3-istio-token\") pod \"141a2288-b7f5-46e6-85cb-c77e432999d3\" (UID: \"141a2288-b7f5-46e6-85cb-c77e432999d3\") "
Oct 30 01:02:13.319976 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.319920    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/141a2288-b7f5-46e6-85cb-c77e432999d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/141a2288-b7f5-46e6-85cb-c77e432999d3-istiod-ca-cert\") pod \"141a2288-b7f5-46e6-85cb-c77e432999d3\" (UID: \"141a2288-b7f5-46e6-85cb-c77e432999d3\") "
Oct 30 01:02:13.320041 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.319971    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/141a2288-b7f5-46e6-85cb-c77e432999d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/141a2288-b7f5-46e6-85cb-c77e432999d3-istio-podinfo\") pod \"141a2288-b7f5-46e6-85cb-c77e432999d3\" (UID: \"141a2288-b7f5-46e6-85cb-c77e432999d3\") "
Oct 30 01:02:13.320041 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.320013    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/141a2288-b7f5-46e6-85cb-c77e432999d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/141a2288-b7f5-46e6-85cb-c77e432999d3-istio-data\") pod \"141a2288-b7f5-46e6-85cb-c77e432999d3\" (UID: \"141a2288-b7f5-46e6-85cb-c77e432999d3\") "
Oct 30 01:02:13.320147 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.320132    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/141a2288-b7f5-46e6-85cb-c77e432999d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/141a2288-b7f5-46e6-85cb-c77e432999d3-workload-socket\") pod \"141a2288-b7f5-46e6-85cb-c77e432999d3\" (UID: \"141a2288-b7f5-46e6-85cb-c77e432999d3\") "
Oct 30 01:02:13.320206 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.320168    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/141a2288-b7f5-46e6-85cb-c77e432999d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/141a2288-b7f5-46e6-85cb-c77e432999d3-credential-socket\") pod \"141a2288-b7f5-46e6-85cb-c77e432999d3\" (UID: \"141a2288-b7f5-46e6-85cb-c77e432999d3\") "
Oct 30 01:02:13.321118 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.320205    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/141a2288-b7f5-46e6-85cb-c77e432999d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/141a2288-b7f5-46e6-85cb-c77e432999d3-workload-certs\") pod \"141a2288-b7f5-46e6-85cb-c77e432999d3\" (UID: \"141a2288-b7f5-46e6-85cb-c77e432999d3\") "
Oct 30 01:02:13.321118 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.320272    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/141a2288-b7f5-46e6-85cb-c77e432999d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/141a2288-b7f5-46e6-85cb-c77e432999d3-istio-envoy\") pod \"141a2288-b7f5-46e6-85cb-c77e432999d3\" (UID: \"141a2288-b7f5-46e6-85cb-c77e432999d3\") "
Oct 30 01:02:13.321118 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.320362    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/141a2288-b7f5-46e6-85cb-c77e432999d3-istiod-ca-cert" pod "141a2288-b7f5-46e6-85cb-c77e432999d3" (UID: "141a2288-b7f5-46e6-85cb-c77e432999d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:02:13.321118 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.320716    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/141a2288-b7f5-46e6-85cb-c77e432999d3-workload-socket" pod "141a2288-b7f5-46e6-85cb-c77e432999d3" (UID: "141a2288-b7f5-46e6-85cb-c77e432999d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:02:13.321118 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.321009    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/141a2288-b7f5-46e6-85cb-c77e432999d3-istio-data" pod "141a2288-b7f5-46e6-85cb-c77e432999d3" (UID: "141a2288-b7f5-46e6-85cb-c77e432999d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:02:13.321555 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.321178    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/141a2288-b7f5-46e6-85cb-c77e432999d3-workload-certs" pod "141a2288-b7f5-46e6-85cb-c77e432999d3" (UID: "141a2288-b7f5-46e6-85cb-c77e432999d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:02:13.321555 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.321364    4393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/141a2288-b7f5-46e6-85cb-c77e432999d3-istiod-ca-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:02:13.321555 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.321387    4393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/141a2288-b7f5-46e6-85cb-c77e432999d3-istio-data\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:02:13.321555 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.321400    4393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/141a2288-b7f5-46e6-85cb-c77e432999d3-workload-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:02:13.321555 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.321413    4393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/141a2288-b7f5-46e6-85cb-c77e432999d3-workload-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:02:13.321978 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.321608    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/141a2288-b7f5-46e6-85cb-c77e432999d3-credential-socket" pod "141a2288-b7f5-46e6-85cb-c77e432999d3" (UID: "141a2288-b7f5-46e6-85cb-c77e432999d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:02:13.323633 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.323608    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/141a2288-b7f5-46e6-85cb-c77e432999d3-kube-api-access-xmtdn" pod "141a2288-b7f5-46e6-85cb-c77e432999d3" (UID: "141a2288-b7f5-46e6-85cb-c77e432999d3"). InnerVolumeSpecName "kube-api-access-xmtdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:02:13.324040 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.324018    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/141a2288-b7f5-46e6-85cb-c77e432999d3-istio-podinfo" pod "141a2288-b7f5-46e6-85cb-c77e432999d3" (UID: "141a2288-b7f5-46e6-85cb-c77e432999d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:02:13.324294 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.324266    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/141a2288-b7f5-46e6-85cb-c77e432999d3-istio-token" pod "141a2288-b7f5-46e6-85cb-c77e432999d3" (UID: "141a2288-b7f5-46e6-85cb-c77e432999d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:02:13.325862 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.325837    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/141a2288-b7f5-46e6-85cb-c77e432999d3-istio-envoy" pod "141a2288-b7f5-46e6-85cb-c77e432999d3" (UID: "141a2288-b7f5-46e6-85cb-c77e432999d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:02:13.422826 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.422790    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmtdn\" (UniqueName: \"kubernetes.io/projected/141a2288-b7f5-46e6-85cb-c77e432999d3-kube-api-access-xmtdn\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:02:13.422826 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.422822    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/141a2288-b7f5-46e6-85cb-c77e432999d3-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:02:13.422826 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.422832    4393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/141a2288-b7f5-46e6-85cb-c77e432999d3-istio-podinfo\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:02:13.423286 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.422842    4393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/141a2288-b7f5-46e6-85cb-c77e432999d3-credential-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:02:13.423286 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.422851    4393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/141a2288-b7f5-46e6-85cb-c77e432999d3-istio-envoy\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:02:13.735556 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.735521    4393 generic.go:356] "Generic (PLEG): container finished" podID="141a2288-b7f5-46e6-85cb-c77e432999d3" containerID="10249610d99c7036916c02e0ffff104f677ffbf06619e8b413c1223498bcd08f" exitCode=0
Oct 30 01:02:13.735754 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.735576    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"
Oct 30 01:02:13.735754 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.735612    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h" event={"ID":"141a2288-b7f5-46e6-85cb-c77e432999d3","Type":"ContainerDied","Data":"10249610d99c7036916c02e0ffff104f677ffbf06619e8b413c1223498bcd08f"}
Oct 30 01:02:13.735754 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.735674    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h" event={"ID":"141a2288-b7f5-46e6-85cb-c77e432999d3","Type":"ContainerDied","Data":"c5e3f4758afa9e58e21ba9a1f27cbd035c44ffd511c132c34a78e953468dc8c8"}
Oct 30 01:02:13.735754 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.735703    4393 scope.go:122] "RemoveContainer" containerID="10249610d99c7036916c02e0ffff104f677ffbf06619e8b413c1223498bcd08f"
Oct 30 01:02:13.747630 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.747607    4393 scope.go:122] "RemoveContainer" containerID="10249610d99c7036916c02e0ffff104f677ffbf06619e8b413c1223498bcd08f"
Oct 30 01:02:13.747971 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:02:13.747946    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10249610d99c7036916c02e0ffff104f677ffbf06619e8b413c1223498bcd08f\": container with ID starting with 10249610d99c7036916c02e0ffff104f677ffbf06619e8b413c1223498bcd08f not found: ID does not exist" containerID="10249610d99c7036916c02e0ffff104f677ffbf06619e8b413c1223498bcd08f"
Oct 30 01:02:13.748057 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.747982    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10249610d99c7036916c02e0ffff104f677ffbf06619e8b413c1223498bcd08f"} err="failed to get container status \"10249610d99c7036916c02e0ffff104f677ffbf06619e8b413c1223498bcd08f\": rpc error: code = NotFound desc = could not find container \"10249610d99c7036916c02e0ffff104f677ffbf06619e8b413c1223498bcd08f\": container with ID starting with 10249610d99c7036916c02e0ffff104f677ffbf06619e8b413c1223498bcd08f not found: ID does not exist"
Oct 30 01:02:13.784986 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.784953    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"]
Oct 30 01:02:13.790678 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:13.790644    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-d2l9h"]
Oct 30 01:02:15.728021 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:02:15.727981    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="141a2288-b7f5-46e6-85cb-c77e432999d3" path="/var/lib/kubelet/pods/141a2288-b7f5-46e6-85cb-c77e432999d3/volumes"
Oct 30 01:02:26.512404 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:02:26.512362    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:02:56.511872 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:02:56.511824    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:03:09.316946 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:09.316916    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm"]
Oct 30 01:03:09.317768 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:09.317747    4393 cpu_manager.go:414] "containerMap: removing container" podUID="141a2288-b7f5-46e6-85cb-c77e432999d3" containerName="istio-proxy"
Oct 30 01:03:09.317768 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:09.317766    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="141a2288-b7f5-46e6-85cb-c77e432999d3" containerName="istio-proxy"
Oct 30 01:03:09.317958 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:09.317816    4393 cpu_manager.go:414] "containerMap: removing container" podUID="62a0449f-5a15-4e71-83aa-49d886d97a79" containerName="main"
Oct 30 01:03:09.317958 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:09.317821    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62a0449f-5a15-4e71-83aa-49d886d97a79" containerName="main"
Oct 30 01:03:09.317958 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:09.317939    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="62a0449f-5a15-4e71-83aa-49d886d97a79" containerName="main"
Oct 30 01:03:09.317958 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:09.317953    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="141a2288-b7f5-46e6-85cb-c77e432999d3" containerName="istio-proxy"
Oct 30 01:03:09.321922 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:09.321905    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm"
Oct 30 01:03:09.341011 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:09.340985    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm"]
Oct 30 01:03:09.344860 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:09.344839    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-19bc68-serving-cert\""
Oct 30 01:03:09.444522 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:09.444480    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77816148-3d88-40c8-8ace-5a71f39dd378-openshift-service-ca\") pod \"maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm\" (UID: \"77816148-3d88-40c8-8ace-5a71f39dd378\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm"
Oct 30 01:03:09.444696 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:09.444621    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77816148-3d88-40c8-8ace-5a71f39dd378-maas-api-tls\") pod \"maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm\" (UID: \"77816148-3d88-40c8-8ace-5a71f39dd378\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm"
Oct 30 01:03:09.444755 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:09.444737    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77816148-3d88-40c8-8ace-5a71f39dd378-metrics-certs\") pod \"maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm\" (UID: \"77816148-3d88-40c8-8ace-5a71f39dd378\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm"
Oct 30 01:03:09.444971 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:09.444943    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9ndl\" (UniqueName: \"kubernetes.io/projected/77816148-3d88-40c8-8ace-5a71f39dd378-kube-api-access-v9ndl\") pod \"maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm\" (UID: \"77816148-3d88-40c8-8ace-5a71f39dd378\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm"
Oct 30 01:03:09.546311 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:09.546268    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77816148-3d88-40c8-8ace-5a71f39dd378-openshift-service-ca\") pod \"maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm\" (UID: \"77816148-3d88-40c8-8ace-5a71f39dd378\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm"
Oct 30 01:03:09.546491 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:09.546368    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77816148-3d88-40c8-8ace-5a71f39dd378-maas-api-tls\") pod \"maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm\" (UID: \"77816148-3d88-40c8-8ace-5a71f39dd378\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm"
Oct 30 01:03:09.546491 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:09.546443    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77816148-3d88-40c8-8ace-5a71f39dd378-metrics-certs\") pod \"maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm\" (UID: \"77816148-3d88-40c8-8ace-5a71f39dd378\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm"
Oct 30 01:03:09.546606 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:09.546590    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9ndl\" (UniqueName: \"kubernetes.io/projected/77816148-3d88-40c8-8ace-5a71f39dd378-kube-api-access-v9ndl\") pod \"maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm\" (UID: \"77816148-3d88-40c8-8ace-5a71f39dd378\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm"
Oct 30 01:03:09.547155 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:09.547125    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77816148-3d88-40c8-8ace-5a71f39dd378-openshift-service-ca\") pod \"maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm\" (UID: \"77816148-3d88-40c8-8ace-5a71f39dd378\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm"
Oct 30 01:03:09.549101 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:09.549069    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77816148-3d88-40c8-8ace-5a71f39dd378-maas-api-tls\") pod \"maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm\" (UID: \"77816148-3d88-40c8-8ace-5a71f39dd378\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm"
Oct 30 01:03:09.549216 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:09.549110    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77816148-3d88-40c8-8ace-5a71f39dd378-metrics-certs\") pod \"maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm\" (UID: \"77816148-3d88-40c8-8ace-5a71f39dd378\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm"
Oct 30 01:03:09.606823 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:09.606716    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9ndl\" (UniqueName: \"kubernetes.io/projected/77816148-3d88-40c8-8ace-5a71f39dd378-kube-api-access-v9ndl\") pod \"maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm\" (UID: \"77816148-3d88-40c8-8ace-5a71f39dd378\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm"
Oct 30 01:03:09.641239 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:09.641191    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm"
Oct 30 01:03:09.990799 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:09.990751    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm"]
Oct 30 01:03:09.991709 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:03:09.991684    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77816148_3d88_40c8_8ace_5a71f39dd378.slice/crio-cdfce79e4f891aa8d693ceb523fa9f432d27c52c7021d60f1eaf56b3668b90b2 WatchSource:0}: Error finding container cdfce79e4f891aa8d693ceb523fa9f432d27c52c7021d60f1eaf56b3668b90b2: Status 404 returned error can't find the container with id cdfce79e4f891aa8d693ceb523fa9f432d27c52c7021d60f1eaf56b3668b90b2
Oct 30 01:03:10.464954 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:10.464921    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm" event={"ID":"77816148-3d88-40c8-8ace-5a71f39dd378","Type":"ContainerStarted","Data":"cdfce79e4f891aa8d693ceb523fa9f432d27c52c7021d60f1eaf56b3668b90b2"}
Oct 30 01:03:11.475103 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:11.475056    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm" event={"ID":"77816148-3d88-40c8-8ace-5a71f39dd378","Type":"ContainerStarted","Data":"694cc7ad9c7fa40e7307101fc3c2ccc8ce3865bcb00c357acb1e7442bc550603"}
Oct 30 01:03:11.475525 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:11.475127    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm"
Oct 30 01:03:11.517264 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:11.517210    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm" podStartSLOduration=2.063645123 podStartE2EDuration="2.517195574s" totalImagesPullingTime="453.550451ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:11.513633085 +0000 UTC m=+1881.901309471" watchObservedRunningTime="2026-10-30 01:03:11.517195574 +0000 UTC m=+1881.904871963"
Oct 30 01:03:17.490251 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:17.490211    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm"
Oct 30 01:03:22.925670 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:22.925638    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"]
Oct 30 01:03:22.926122 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:22.925911    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn" podUID="e50cc2ce-0447-448d-9c9f-b93131d3fcba" containerName="istio-proxy" containerID="cri-o://743ec3f151e664d2a4f0d686f1da3f1749167f28f42bff20696ed133145f0ad3" gracePeriod=30
Oct 30 01:03:23.746055 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:23.746024    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"]
Oct 30 01:03:23.746254 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:23.746229    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz" podUID="75776f35-e39d-4823-ac2a-61e5b4e69619" containerName="istio-proxy" containerID="cri-o://21274a5260f967854a3e02669bdef052286a44ce05fe8daa5da3648ff98e5841" gracePeriod=30
Oct 30 01:03:26.567308 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:03:26.567259    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:03:28.236328 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.236303    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"
Oct 30 01:03:28.343651 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.343609    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e50cc2ce-0447-448d-9c9f-b93131d3fcba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e50cc2ce-0447-448d-9c9f-b93131d3fcba-istio-data\") pod \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\" (UID: \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\") "
Oct 30 01:03:28.343907 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.343796    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e50cc2ce-0447-448d-9c9f-b93131d3fcba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e50cc2ce-0447-448d-9c9f-b93131d3fcba-credential-socket\") pod \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\" (UID: \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\") "
Oct 30 01:03:28.343992 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.343947    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e50cc2ce-0447-448d-9c9f-b93131d3fcba-istio-data" pod "e50cc2ce-0447-448d-9c9f-b93131d3fcba" (UID: "e50cc2ce-0447-448d-9c9f-b93131d3fcba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:28.343992 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.343970    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e50cc2ce-0447-448d-9c9f-b93131d3fcba-kube-api-access-qpkwq\" (UniqueName: \"kubernetes.io/projected/e50cc2ce-0447-448d-9c9f-b93131d3fcba-kube-api-access-qpkwq\") pod \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\" (UID: \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\") "
Oct 30 01:03:28.344110 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.343999    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e50cc2ce-0447-448d-9c9f-b93131d3fcba-istio-token\" (UniqueName: \"kubernetes.io/projected/e50cc2ce-0447-448d-9c9f-b93131d3fcba-istio-token\") pod \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\" (UID: \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\") "
Oct 30 01:03:28.344110 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.344069    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e50cc2ce-0447-448d-9c9f-b93131d3fcba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e50cc2ce-0447-448d-9c9f-b93131d3fcba-workload-certs\") pod \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\" (UID: \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\") "
Oct 30 01:03:28.344110 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.344103    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e50cc2ce-0447-448d-9c9f-b93131d3fcba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e50cc2ce-0447-448d-9c9f-b93131d3fcba-workload-socket\") pod \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\" (UID: \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\") "
Oct 30 01:03:28.344260 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.344109    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e50cc2ce-0447-448d-9c9f-b93131d3fcba-credential-socket" pod "e50cc2ce-0447-448d-9c9f-b93131d3fcba" (UID: "e50cc2ce-0447-448d-9c9f-b93131d3fcba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:28.344260 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.344146    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e50cc2ce-0447-448d-9c9f-b93131d3fcba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e50cc2ce-0447-448d-9c9f-b93131d3fcba-istio-podinfo\") pod \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\" (UID: \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\") "
Oct 30 01:03:28.344260 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.344195    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e50cc2ce-0447-448d-9c9f-b93131d3fcba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e50cc2ce-0447-448d-9c9f-b93131d3fcba-istiod-ca-cert\") pod \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\" (UID: \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\") "
Oct 30 01:03:28.344428 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.344309    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e50cc2ce-0447-448d-9c9f-b93131d3fcba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e50cc2ce-0447-448d-9c9f-b93131d3fcba-istio-envoy\") pod \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\" (UID: \"e50cc2ce-0447-448d-9c9f-b93131d3fcba\") "
Oct 30 01:03:28.344428 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.344374    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e50cc2ce-0447-448d-9c9f-b93131d3fcba-workload-certs" pod "e50cc2ce-0447-448d-9c9f-b93131d3fcba" (UID: "e50cc2ce-0447-448d-9c9f-b93131d3fcba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:28.344939 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.344676    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e50cc2ce-0447-448d-9c9f-b93131d3fcba-istiod-ca-cert" pod "e50cc2ce-0447-448d-9c9f-b93131d3fcba" (UID: "e50cc2ce-0447-448d-9c9f-b93131d3fcba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:03:28.345308 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.345286    4393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e50cc2ce-0447-448d-9c9f-b93131d3fcba-credential-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:03:28.345464 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.345448    4393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e50cc2ce-0447-448d-9c9f-b93131d3fcba-workload-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:03:28.345597 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.345583    4393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e50cc2ce-0447-448d-9c9f-b93131d3fcba-istiod-ca-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:03:28.345751 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.345739    4393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e50cc2ce-0447-448d-9c9f-b93131d3fcba-istio-data\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:03:28.345894 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.345292    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e50cc2ce-0447-448d-9c9f-b93131d3fcba-workload-socket" pod "e50cc2ce-0447-448d-9c9f-b93131d3fcba" (UID: "e50cc2ce-0447-448d-9c9f-b93131d3fcba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:28.347060 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.347032    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e50cc2ce-0447-448d-9c9f-b93131d3fcba-istio-token" pod "e50cc2ce-0447-448d-9c9f-b93131d3fcba" (UID: "e50cc2ce-0447-448d-9c9f-b93131d3fcba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:28.347165 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.347135    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e50cc2ce-0447-448d-9c9f-b93131d3fcba-kube-api-access-qpkwq" pod "e50cc2ce-0447-448d-9c9f-b93131d3fcba" (UID: "e50cc2ce-0447-448d-9c9f-b93131d3fcba"). InnerVolumeSpecName "kube-api-access-qpkwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:28.347600 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.347576    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e50cc2ce-0447-448d-9c9f-b93131d3fcba-istio-podinfo" pod "e50cc2ce-0447-448d-9c9f-b93131d3fcba" (UID: "e50cc2ce-0447-448d-9c9f-b93131d3fcba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:03:28.347704 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.347664    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e50cc2ce-0447-448d-9c9f-b93131d3fcba-istio-envoy" pod "e50cc2ce-0447-448d-9c9f-b93131d3fcba" (UID: "e50cc2ce-0447-448d-9c9f-b93131d3fcba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:28.448038 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.447961    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpkwq\" (UniqueName: \"kubernetes.io/projected/e50cc2ce-0447-448d-9c9f-b93131d3fcba-kube-api-access-qpkwq\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:03:28.448038 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.447992    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e50cc2ce-0447-448d-9c9f-b93131d3fcba-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:03:28.448038 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.448002    4393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e50cc2ce-0447-448d-9c9f-b93131d3fcba-workload-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:03:28.448038 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.448010    4393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e50cc2ce-0447-448d-9c9f-b93131d3fcba-istio-podinfo\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:03:28.448038 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.448018    4393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e50cc2ce-0447-448d-9c9f-b93131d3fcba-istio-envoy\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:03:28.662426 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.662388    4393 generic.go:356] "Generic (PLEG): container finished" podID="e50cc2ce-0447-448d-9c9f-b93131d3fcba" containerID="743ec3f151e664d2a4f0d686f1da3f1749167f28f42bff20696ed133145f0ad3" exitCode=0
Oct 30 01:03:28.662640 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.662442    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"
Oct 30 01:03:28.662640 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.662476    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn" event={"ID":"e50cc2ce-0447-448d-9c9f-b93131d3fcba","Type":"ContainerDied","Data":"743ec3f151e664d2a4f0d686f1da3f1749167f28f42bff20696ed133145f0ad3"}
Oct 30 01:03:28.662640 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.662544    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn" event={"ID":"e50cc2ce-0447-448d-9c9f-b93131d3fcba","Type":"ContainerDied","Data":"cbb8f58b7a775516d9147d03bdc87714125ea84e5569dd10729c87b25988e048"}
Oct 30 01:03:28.662640 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.662564    4393 scope.go:122] "RemoveContainer" containerID="743ec3f151e664d2a4f0d686f1da3f1749167f28f42bff20696ed133145f0ad3"
Oct 30 01:03:28.677589 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.677565    4393 scope.go:122] "RemoveContainer" containerID="743ec3f151e664d2a4f0d686f1da3f1749167f28f42bff20696ed133145f0ad3"
Oct 30 01:03:28.677922 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:03:28.677903    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"743ec3f151e664d2a4f0d686f1da3f1749167f28f42bff20696ed133145f0ad3\": container with ID starting with 743ec3f151e664d2a4f0d686f1da3f1749167f28f42bff20696ed133145f0ad3 not found: ID does not exist" containerID="743ec3f151e664d2a4f0d686f1da3f1749167f28f42bff20696ed133145f0ad3"
Oct 30 01:03:28.677988 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.677934    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"743ec3f151e664d2a4f0d686f1da3f1749167f28f42bff20696ed133145f0ad3"} err="failed to get container status \"743ec3f151e664d2a4f0d686f1da3f1749167f28f42bff20696ed133145f0ad3\": rpc error: code = NotFound desc = could not find container \"743ec3f151e664d2a4f0d686f1da3f1749167f28f42bff20696ed133145f0ad3\": container with ID starting with 743ec3f151e664d2a4f0d686f1da3f1749167f28f42bff20696ed133145f0ad3 not found: ID does not exist"
Oct 30 01:03:28.731858 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.731738    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"]
Oct 30 01:03:28.738084 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:28.738057    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-kcwdn"]
Oct 30 01:03:29.085493 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.085470    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"
Oct 30 01:03:29.159602 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.159546    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75776f35-e39d-4823-ac2a-61e5b4e69619-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75776f35-e39d-4823-ac2a-61e5b4e69619-workload-certs\") pod \"75776f35-e39d-4823-ac2a-61e5b4e69619\" (UID: \"75776f35-e39d-4823-ac2a-61e5b4e69619\") "
Oct 30 01:03:29.159790 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.159664    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75776f35-e39d-4823-ac2a-61e5b4e69619-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75776f35-e39d-4823-ac2a-61e5b4e69619-credential-socket\") pod \"75776f35-e39d-4823-ac2a-61e5b4e69619\" (UID: \"75776f35-e39d-4823-ac2a-61e5b4e69619\") "
Oct 30 01:03:29.159790 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.159699    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75776f35-e39d-4823-ac2a-61e5b4e69619-istio-token\" (UniqueName: \"kubernetes.io/projected/75776f35-e39d-4823-ac2a-61e5b4e69619-istio-token\") pod \"75776f35-e39d-4823-ac2a-61e5b4e69619\" (UID: \"75776f35-e39d-4823-ac2a-61e5b4e69619\") "
Oct 30 01:03:29.159878 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.159812    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75776f35-e39d-4823-ac2a-61e5b4e69619-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75776f35-e39d-4823-ac2a-61e5b4e69619-istiod-ca-cert\") pod \"75776f35-e39d-4823-ac2a-61e5b4e69619\" (UID: \"75776f35-e39d-4823-ac2a-61e5b4e69619\") "
Oct 30 01:03:29.159939 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.159870    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75776f35-e39d-4823-ac2a-61e5b4e69619-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75776f35-e39d-4823-ac2a-61e5b4e69619-istio-podinfo\") pod \"75776f35-e39d-4823-ac2a-61e5b4e69619\" (UID: \"75776f35-e39d-4823-ac2a-61e5b4e69619\") "
Oct 30 01:03:29.159997 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.159945    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75776f35-e39d-4823-ac2a-61e5b4e69619-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75776f35-e39d-4823-ac2a-61e5b4e69619-workload-socket\") pod \"75776f35-e39d-4823-ac2a-61e5b4e69619\" (UID: \"75776f35-e39d-4823-ac2a-61e5b4e69619\") "
Oct 30 01:03:29.159997 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.159869    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75776f35-e39d-4823-ac2a-61e5b4e69619-workload-certs" pod "75776f35-e39d-4823-ac2a-61e5b4e69619" (UID: "75776f35-e39d-4823-ac2a-61e5b4e69619"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:29.160102 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.159955    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75776f35-e39d-4823-ac2a-61e5b4e69619-credential-socket" pod "75776f35-e39d-4823-ac2a-61e5b4e69619" (UID: "75776f35-e39d-4823-ac2a-61e5b4e69619"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:29.160102 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.160003    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75776f35-e39d-4823-ac2a-61e5b4e69619-kube-api-access-59s52\" (UniqueName: \"kubernetes.io/projected/75776f35-e39d-4823-ac2a-61e5b4e69619-kube-api-access-59s52\") pod \"75776f35-e39d-4823-ac2a-61e5b4e69619\" (UID: \"75776f35-e39d-4823-ac2a-61e5b4e69619\") "
Oct 30 01:03:29.160102 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.160062    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75776f35-e39d-4823-ac2a-61e5b4e69619-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75776f35-e39d-4823-ac2a-61e5b4e69619-istio-envoy\") pod \"75776f35-e39d-4823-ac2a-61e5b4e69619\" (UID: \"75776f35-e39d-4823-ac2a-61e5b4e69619\") "
Oct 30 01:03:29.160271 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.160229    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75776f35-e39d-4823-ac2a-61e5b4e69619-workload-socket" pod "75776f35-e39d-4823-ac2a-61e5b4e69619" (UID: "75776f35-e39d-4823-ac2a-61e5b4e69619"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:29.160271 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.160236    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75776f35-e39d-4823-ac2a-61e5b4e69619-istiod-ca-cert" pod "75776f35-e39d-4823-ac2a-61e5b4e69619" (UID: "75776f35-e39d-4823-ac2a-61e5b4e69619"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:03:29.160271 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.160235    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75776f35-e39d-4823-ac2a-61e5b4e69619-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75776f35-e39d-4823-ac2a-61e5b4e69619-istio-data\") pod \"75776f35-e39d-4823-ac2a-61e5b4e69619\" (UID: \"75776f35-e39d-4823-ac2a-61e5b4e69619\") "
Oct 30 01:03:29.160612 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.160570    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75776f35-e39d-4823-ac2a-61e5b4e69619-istio-data" pod "75776f35-e39d-4823-ac2a-61e5b4e69619" (UID: "75776f35-e39d-4823-ac2a-61e5b4e69619"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:29.161345 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.161321    4393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75776f35-e39d-4823-ac2a-61e5b4e69619-istio-data\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:03:29.161453 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.161349    4393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75776f35-e39d-4823-ac2a-61e5b4e69619-workload-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:03:29.161453 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.161364    4393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75776f35-e39d-4823-ac2a-61e5b4e69619-credential-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:03:29.161453 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.161377    4393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75776f35-e39d-4823-ac2a-61e5b4e69619-istiod-ca-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:03:29.161453 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.161390    4393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75776f35-e39d-4823-ac2a-61e5b4e69619-workload-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:03:29.164498 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.162484    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75776f35-e39d-4823-ac2a-61e5b4e69619-istio-token" pod "75776f35-e39d-4823-ac2a-61e5b4e69619" (UID: "75776f35-e39d-4823-ac2a-61e5b4e69619"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:29.164498 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.162751    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75776f35-e39d-4823-ac2a-61e5b4e69619-istio-podinfo" pod "75776f35-e39d-4823-ac2a-61e5b4e69619" (UID: "75776f35-e39d-4823-ac2a-61e5b4e69619"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:03:29.164498 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.162770    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75776f35-e39d-4823-ac2a-61e5b4e69619-kube-api-access-59s52" pod "75776f35-e39d-4823-ac2a-61e5b4e69619" (UID: "75776f35-e39d-4823-ac2a-61e5b4e69619"). InnerVolumeSpecName "kube-api-access-59s52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:29.164498 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.162827    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75776f35-e39d-4823-ac2a-61e5b4e69619-istio-envoy" pod "75776f35-e39d-4823-ac2a-61e5b4e69619" (UID: "75776f35-e39d-4823-ac2a-61e5b4e69619"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:03:29.263599 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.263431    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75776f35-e39d-4823-ac2a-61e5b4e69619-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:03:29.263599 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.263471    4393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75776f35-e39d-4823-ac2a-61e5b4e69619-istio-podinfo\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:03:29.263599 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.263485    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59s52\" (UniqueName: \"kubernetes.io/projected/75776f35-e39d-4823-ac2a-61e5b4e69619-kube-api-access-59s52\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:03:29.263599 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.263494    4393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75776f35-e39d-4823-ac2a-61e5b4e69619-istio-envoy\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:03:29.672590 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.672554    4393 generic.go:356] "Generic (PLEG): container finished" podID="75776f35-e39d-4823-ac2a-61e5b4e69619" containerID="21274a5260f967854a3e02669bdef052286a44ce05fe8daa5da3648ff98e5841" exitCode=0
Oct 30 01:03:29.672802 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.672611    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"
Oct 30 01:03:29.672802 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.672645    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz" event={"ID":"75776f35-e39d-4823-ac2a-61e5b4e69619","Type":"ContainerDied","Data":"21274a5260f967854a3e02669bdef052286a44ce05fe8daa5da3648ff98e5841"}
Oct 30 01:03:29.672802 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.672703    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz" event={"ID":"75776f35-e39d-4823-ac2a-61e5b4e69619","Type":"ContainerDied","Data":"b4f7a46546a1c174e4a1c619704db20e4d456e34692ec0ed73f31cf2d3fb829e"}
Oct 30 01:03:29.672802 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.672731    4393 scope.go:122] "RemoveContainer" containerID="21274a5260f967854a3e02669bdef052286a44ce05fe8daa5da3648ff98e5841"
Oct 30 01:03:29.684898 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.684875    4393 scope.go:122] "RemoveContainer" containerID="21274a5260f967854a3e02669bdef052286a44ce05fe8daa5da3648ff98e5841"
Oct 30 01:03:29.685186 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:03:29.685166    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21274a5260f967854a3e02669bdef052286a44ce05fe8daa5da3648ff98e5841\": container with ID starting with 21274a5260f967854a3e02669bdef052286a44ce05fe8daa5da3648ff98e5841 not found: ID does not exist" containerID="21274a5260f967854a3e02669bdef052286a44ce05fe8daa5da3648ff98e5841"
Oct 30 01:03:29.685261 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.685198    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21274a5260f967854a3e02669bdef052286a44ce05fe8daa5da3648ff98e5841"} err="failed to get container status \"21274a5260f967854a3e02669bdef052286a44ce05fe8daa5da3648ff98e5841\": rpc error: code = NotFound desc = could not find container \"21274a5260f967854a3e02669bdef052286a44ce05fe8daa5da3648ff98e5841\": container with ID starting with 21274a5260f967854a3e02669bdef052286a44ce05fe8daa5da3648ff98e5841 not found: ID does not exist"
Oct 30 01:03:29.733961 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.733928    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e50cc2ce-0447-448d-9c9f-b93131d3fcba" path="/var/lib/kubelet/pods/e50cc2ce-0447-448d-9c9f-b93131d3fcba/volumes"
Oct 30 01:03:29.734392 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.734377    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"]
Oct 30 01:03:29.737878 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:29.737856    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-4mggz"]
Oct 30 01:03:31.728931 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:31.728895    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75776f35-e39d-4823-ac2a-61e5b4e69619" path="/var/lib/kubelet/pods/75776f35-e39d-4823-ac2a-61e5b4e69619/volumes"
Oct 30 01:03:36.368807 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:36.364846    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv"]
Oct 30 01:03:36.368807 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:36.366209    4393 cpu_manager.go:414] "containerMap: removing container" podUID="e50cc2ce-0447-448d-9c9f-b93131d3fcba" containerName="istio-proxy"
Oct 30 01:03:36.368807 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:36.366232    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e50cc2ce-0447-448d-9c9f-b93131d3fcba" containerName="istio-proxy"
Oct 30 01:03:36.368807 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:36.366267    4393 cpu_manager.go:414] "containerMap: removing container" podUID="75776f35-e39d-4823-ac2a-61e5b4e69619" containerName="istio-proxy"
Oct 30 01:03:36.368807 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:36.366275    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75776f35-e39d-4823-ac2a-61e5b4e69619" containerName="istio-proxy"
Oct 30 01:03:36.368807 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:36.366522    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="e50cc2ce-0447-448d-9c9f-b93131d3fcba" containerName="istio-proxy"
Oct 30 01:03:36.368807 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:36.366546    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="75776f35-e39d-4823-ac2a-61e5b4e69619" containerName="istio-proxy"
Oct 30 01:03:36.392804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:36.385588    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv"
Oct 30 01:03:36.453811 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:36.439548    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv"]
Oct 30 01:03:36.459857 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:36.458375    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08c5615b-d854-4dd7-843e-9764dab5a127-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv\" (UID: \"08c5615b-d854-4dd7-843e-9764dab5a127\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv"
Oct 30 01:03:36.459857 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:36.458627    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzwr8\" (UniqueName: \"kubernetes.io/projected/08c5615b-d854-4dd7-843e-9764dab5a127-kube-api-access-jzwr8\") pod \"maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv\" (UID: \"08c5615b-d854-4dd7-843e-9764dab5a127\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv"
Oct 30 01:03:36.459857 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:36.458696    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08c5615b-d854-4dd7-843e-9764dab5a127-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv\" (UID: \"08c5615b-d854-4dd7-843e-9764dab5a127\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv"
Oct 30 01:03:36.459857 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:36.458754    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08c5615b-d854-4dd7-843e-9764dab5a127-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv\" (UID: \"08c5615b-d854-4dd7-843e-9764dab5a127\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv"
Oct 30 01:03:36.511804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:36.511205    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-beeba4-serving-cert\""
Oct 30 01:03:36.559970 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:36.559933    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08c5615b-d854-4dd7-843e-9764dab5a127-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv\" (UID: \"08c5615b-d854-4dd7-843e-9764dab5a127\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv"
Oct 30 01:03:36.560119 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:36.560003    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08c5615b-d854-4dd7-843e-9764dab5a127-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv\" (UID: \"08c5615b-d854-4dd7-843e-9764dab5a127\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv"
Oct 30 01:03:36.560178 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:03:36.560115    4393 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-beeba4-serving-cert: secret "maas-api-e2e-shared-a-w4-beeba4-serving-cert" not found
Oct 30 01:03:36.560178 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:36.560165    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08c5615b-d854-4dd7-843e-9764dab5a127-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv\" (UID: \"08c5615b-d854-4dd7-843e-9764dab5a127\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv"
Oct 30 01:03:36.560261 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:03:36.560207    4393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08c5615b-d854-4dd7-843e-9764dab5a127-maas-api-tls podName:08c5615b-d854-4dd7-843e-9764dab5a127 nodeName:}" failed. No retries permitted until 2026-10-30 01:03:37.060183014 +0000 UTC m=+1907.447859404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/08c5615b-d854-4dd7-843e-9764dab5a127-maas-api-tls") pod "maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv" (UID: "08c5615b-d854-4dd7-843e-9764dab5a127") : secret "maas-api-e2e-shared-a-w4-beeba4-serving-cert" not found
Oct 30 01:03:36.560501 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:36.560479    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzwr8\" (UniqueName: \"kubernetes.io/projected/08c5615b-d854-4dd7-843e-9764dab5a127-kube-api-access-jzwr8\") pod \"maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv\" (UID: \"08c5615b-d854-4dd7-843e-9764dab5a127\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv"
Oct 30 01:03:36.560808 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:36.560749    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08c5615b-d854-4dd7-843e-9764dab5a127-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv\" (UID: \"08c5615b-d854-4dd7-843e-9764dab5a127\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv"
Oct 30 01:03:36.562710 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:36.562685    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08c5615b-d854-4dd7-843e-9764dab5a127-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv\" (UID: \"08c5615b-d854-4dd7-843e-9764dab5a127\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv"
Oct 30 01:03:36.585748 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:36.585709    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzwr8\" (UniqueName: \"kubernetes.io/projected/08c5615b-d854-4dd7-843e-9764dab5a127-kube-api-access-jzwr8\") pod \"maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv\" (UID: \"08c5615b-d854-4dd7-843e-9764dab5a127\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv"
Oct 30 01:03:37.099630 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:37.088858    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08c5615b-d854-4dd7-843e-9764dab5a127-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv\" (UID: \"08c5615b-d854-4dd7-843e-9764dab5a127\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv"
Oct 30 01:03:37.100514 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:37.100450    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08c5615b-d854-4dd7-843e-9764dab5a127-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv\" (UID: \"08c5615b-d854-4dd7-843e-9764dab5a127\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv"
Oct 30 01:03:37.131932 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:37.131593    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv"
Oct 30 01:03:38.090059 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:38.089903    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv"]
Oct 30 01:03:38.362388 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:38.362293    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf"]
Oct 30 01:03:38.362570 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:38.362534    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf" podUID="b7357139-9713-42ac-bdbe-c9093760765b" containerName="maas-api" containerID="cri-o://03fc326520385a5bfe13f7785cdda51c141845a8c12d76ad348c1c6115edf429" gracePeriod=30
Oct 30 01:03:38.681525 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:38.681491    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf"
Oct 30 01:03:38.795870 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:38.790652    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7357139-9713-42ac-bdbe-c9093760765b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7357139-9713-42ac-bdbe-c9093760765b-metrics-certs\") pod \"b7357139-9713-42ac-bdbe-c9093760765b\" (UID: \"b7357139-9713-42ac-bdbe-c9093760765b\") "
Oct 30 01:03:38.795870 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:38.790800    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7357139-9713-42ac-bdbe-c9093760765b-kube-api-access-sv9g2\" (UniqueName: \"kubernetes.io/projected/b7357139-9713-42ac-bdbe-c9093760765b-kube-api-access-sv9g2\") pod \"b7357139-9713-42ac-bdbe-c9093760765b\" (UID: \"b7357139-9713-42ac-bdbe-c9093760765b\") "
Oct 30 01:03:38.795870 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:38.790874    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7357139-9713-42ac-bdbe-c9093760765b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7357139-9713-42ac-bdbe-c9093760765b-maas-api-tls\") pod \"b7357139-9713-42ac-bdbe-c9093760765b\" (UID: \"b7357139-9713-42ac-bdbe-c9093760765b\") "
Oct 30 01:03:38.795870 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:38.790920    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7357139-9713-42ac-bdbe-c9093760765b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7357139-9713-42ac-bdbe-c9093760765b-openshift-service-ca\") pod \"b7357139-9713-42ac-bdbe-c9093760765b\" (UID: \"b7357139-9713-42ac-bdbe-c9093760765b\") "
Oct 30 01:03:38.809517 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:38.809474    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7357139-9713-42ac-bdbe-c9093760765b-openshift-service-ca" pod "b7357139-9713-42ac-bdbe-c9093760765b" (UID: "b7357139-9713-42ac-bdbe-c9093760765b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:03:38.856820 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:38.842710    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7357139-9713-42ac-bdbe-c9093760765b-maas-api-tls" pod "b7357139-9713-42ac-bdbe-c9093760765b" (UID: "b7357139-9713-42ac-bdbe-c9093760765b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:03:38.856820 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:38.854205    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7357139-9713-42ac-bdbe-c9093760765b-metrics-certs" pod "b7357139-9713-42ac-bdbe-c9093760765b" (UID: "b7357139-9713-42ac-bdbe-c9093760765b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:03:38.875808 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:38.864277    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7357139-9713-42ac-bdbe-c9093760765b-kube-api-access-sv9g2" pod "b7357139-9713-42ac-bdbe-c9093760765b" (UID: "b7357139-9713-42ac-bdbe-c9093760765b"). InnerVolumeSpecName "kube-api-access-sv9g2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:03:38.901909 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:38.901598    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7357139-9713-42ac-bdbe-c9093760765b-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:03:38.901909 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:38.901637    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv9g2\" (UniqueName: \"kubernetes.io/projected/b7357139-9713-42ac-bdbe-c9093760765b-kube-api-access-sv9g2\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:03:38.901909 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:38.901651    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7357139-9713-42ac-bdbe-c9093760765b-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:03:38.901909 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:38.901665    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7357139-9713-42ac-bdbe-c9093760765b-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:03:39.165285 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:39.165193    4393 generic.go:356] "Generic (PLEG): container finished" podID="b7357139-9713-42ac-bdbe-c9093760765b" containerID="03fc326520385a5bfe13f7785cdda51c141845a8c12d76ad348c1c6115edf429" exitCode=0
Oct 30 01:03:39.187017 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:39.169937    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf"
Oct 30 01:03:39.222812 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:39.169984    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf" event={"ID":"b7357139-9713-42ac-bdbe-c9093760765b","Type":"ContainerDied","Data":"03fc326520385a5bfe13f7785cdda51c141845a8c12d76ad348c1c6115edf429"}
Oct 30 01:03:39.222812 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:39.208282    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf" event={"ID":"b7357139-9713-42ac-bdbe-c9093760765b","Type":"ContainerDied","Data":"ce90caf9ee63bf61889d4dacfd191be87ec33cca103f4fea92bc8fcfe7dafa61"}
Oct 30 01:03:39.222812 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:39.208327    4393 scope.go:122] "RemoveContainer" containerID="03fc326520385a5bfe13f7785cdda51c141845a8c12d76ad348c1c6115edf429"
Oct 30 01:03:39.222812 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:39.210192    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv" event={"ID":"08c5615b-d854-4dd7-843e-9764dab5a127","Type":"ContainerStarted","Data":"c5b5b1cd585117146b93df782c21c9d0afd11d1bf3a84558455c9f2418ddf72d"}
Oct 30 01:03:39.222812 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:39.210243    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv" event={"ID":"08c5615b-d854-4dd7-843e-9764dab5a127","Type":"ContainerStarted","Data":"1c94c833804b5899797c127a8610e83d004afa448970eb3ef954e68d583bb48b"}
Oct 30 01:03:39.222812 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:39.211481    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv"
Oct 30 01:03:39.233595 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:39.233574    4393 scope.go:122] "RemoveContainer" containerID="03fc326520385a5bfe13f7785cdda51c141845a8c12d76ad348c1c6115edf429"
Oct 30 01:03:39.234029 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:03:39.234010    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03fc326520385a5bfe13f7785cdda51c141845a8c12d76ad348c1c6115edf429\": container with ID starting with 03fc326520385a5bfe13f7785cdda51c141845a8c12d76ad348c1c6115edf429 not found: ID does not exist" containerID="03fc326520385a5bfe13f7785cdda51c141845a8c12d76ad348c1c6115edf429"
Oct 30 01:03:39.234152 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:39.234134    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03fc326520385a5bfe13f7785cdda51c141845a8c12d76ad348c1c6115edf429"} err="failed to get container status \"03fc326520385a5bfe13f7785cdda51c141845a8c12d76ad348c1c6115edf429\": rpc error: code = NotFound desc = could not find container \"03fc326520385a5bfe13f7785cdda51c141845a8c12d76ad348c1c6115edf429\": container with ID starting with 03fc326520385a5bfe13f7785cdda51c141845a8c12d76ad348c1c6115edf429 not found: ID does not exist"
Oct 30 01:03:39.259721 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:39.259528    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv" podStartSLOduration=2.868322173 podStartE2EDuration="3.259513947s" totalImagesPullingTime="391.191774ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:03:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:03:39.254608767 +0000 UTC m=+1909.642285155" watchObservedRunningTime="2026-10-30 01:03:39.259513947 +0000 UTC m=+1909.647190365"
Oct 30 01:03:39.307870 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:39.307835    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf"]
Oct 30 01:03:39.313040 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:39.313017    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-481ab7-7b9d5f44d8-2wvnf"]
Oct 30 01:03:39.727985 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:39.727952    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7357139-9713-42ac-bdbe-c9093760765b" path="/var/lib/kubelet/pods/b7357139-9713-42ac-bdbe-c9093760765b/volumes"
Oct 30 01:03:46.236988 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:46.236952    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv"
Oct 30 01:03:50.676032 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:03:50.675996    4393 scope.go:122] "RemoveContainer" containerID="eb158a72650e6f0c36a02a50c655ed185ccd0fcb7498c549e86985ecfabf503e"
Oct 30 01:03:56.514910 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:03:56.514870    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:04:23.055872 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:23.055828    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"]
Oct 30 01:04:23.057003 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:23.056137    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8" podUID="eb8b407b-2bc7-4457-814b-31ffbdffa0c2" containerName="istio-proxy" containerID="cri-o://4c5c8f964a98d054cfc3f0d40790fc145fc1ca59e5bb5632b1b362282cecf9d9" gracePeriod=30
Oct 30 01:04:23.580486 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:23.580452    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"]
Oct 30 01:04:23.580674 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:23.580652    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm" podUID="eead8403-b3bd-4a6e-bfc0-0c1785eae339" containerName="istio-proxy" containerID="cri-o://7d5e90364f94a15744f1612d3b0c3b402ad4aa16e75c3d21687784f34c0f0bad" gracePeriod=30
Oct 30 01:04:25.889530 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:25.889488    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2"]
Oct 30 01:04:26.539436 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:04:26.539398    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:04:26.691598 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:26.691557    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2" podUID="016c0a87-3f64-4eb2-ad53-925be83cdf21" containerName="maas-api" containerID="cri-o://88e56eec75a179054b6f9cbe6461eeab23eaf2174766c8f58ac678570f367365" gracePeriod=30
Oct 30 01:04:27.206368 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:27.206336    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2"
Oct 30 01:04:27.395041 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:27.395004    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/016c0a87-3f64-4eb2-ad53-925be83cdf21-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016c0a87-3f64-4eb2-ad53-925be83cdf21-maas-api-tls\") pod \"016c0a87-3f64-4eb2-ad53-925be83cdf21\" (UID: \"016c0a87-3f64-4eb2-ad53-925be83cdf21\") "
Oct 30 01:04:27.395267 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:27.395144    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/016c0a87-3f64-4eb2-ad53-925be83cdf21-kube-api-access-jw7vd\" (UniqueName: \"kubernetes.io/projected/016c0a87-3f64-4eb2-ad53-925be83cdf21-kube-api-access-jw7vd\") pod \"016c0a87-3f64-4eb2-ad53-925be83cdf21\" (UID: \"016c0a87-3f64-4eb2-ad53-925be83cdf21\") "
Oct 30 01:04:27.395506 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:27.395480    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/016c0a87-3f64-4eb2-ad53-925be83cdf21-metrics-certs\" (UniqueName: \"kubernetes.io/secret/016c0a87-3f64-4eb2-ad53-925be83cdf21-metrics-certs\") pod \"016c0a87-3f64-4eb2-ad53-925be83cdf21\" (UID: \"016c0a87-3f64-4eb2-ad53-925be83cdf21\") "
Oct 30 01:04:27.395615 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:27.395541    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/016c0a87-3f64-4eb2-ad53-925be83cdf21-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/016c0a87-3f64-4eb2-ad53-925be83cdf21-openshift-service-ca\") pod \"016c0a87-3f64-4eb2-ad53-925be83cdf21\" (UID: \"016c0a87-3f64-4eb2-ad53-925be83cdf21\") "
Oct 30 01:04:27.396322 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:27.396176    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/016c0a87-3f64-4eb2-ad53-925be83cdf21-openshift-service-ca" pod "016c0a87-3f64-4eb2-ad53-925be83cdf21" (UID: "016c0a87-3f64-4eb2-ad53-925be83cdf21"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:04:27.396464 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:27.396381    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/016c0a87-3f64-4eb2-ad53-925be83cdf21-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:04:27.397876 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:27.397846    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/016c0a87-3f64-4eb2-ad53-925be83cdf21-maas-api-tls" pod "016c0a87-3f64-4eb2-ad53-925be83cdf21" (UID: "016c0a87-3f64-4eb2-ad53-925be83cdf21"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:04:27.398179 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:27.398160    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/016c0a87-3f64-4eb2-ad53-925be83cdf21-kube-api-access-jw7vd" pod "016c0a87-3f64-4eb2-ad53-925be83cdf21" (UID: "016c0a87-3f64-4eb2-ad53-925be83cdf21"). InnerVolumeSpecName "kube-api-access-jw7vd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:27.398323 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:27.398296    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/016c0a87-3f64-4eb2-ad53-925be83cdf21-metrics-certs" pod "016c0a87-3f64-4eb2-ad53-925be83cdf21" (UID: "016c0a87-3f64-4eb2-ad53-925be83cdf21"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:04:27.498565 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:27.498472    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016c0a87-3f64-4eb2-ad53-925be83cdf21-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:04:27.498565 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:27.498502    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw7vd\" (UniqueName: \"kubernetes.io/projected/016c0a87-3f64-4eb2-ad53-925be83cdf21-kube-api-access-jw7vd\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:04:27.498565 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:27.498511    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/016c0a87-3f64-4eb2-ad53-925be83cdf21-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:04:27.729805 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:27.729095    4393 generic.go:356] "Generic (PLEG): container finished" podID="016c0a87-3f64-4eb2-ad53-925be83cdf21" containerID="88e56eec75a179054b6f9cbe6461eeab23eaf2174766c8f58ac678570f367365" exitCode=0
Oct 30 01:04:27.729805 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:27.729318    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2"
Oct 30 01:04:27.747269 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:27.747239    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2" event={"ID":"016c0a87-3f64-4eb2-ad53-925be83cdf21","Type":"ContainerDied","Data":"88e56eec75a179054b6f9cbe6461eeab23eaf2174766c8f58ac678570f367365"}
Oct 30 01:04:27.747418 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:27.747303    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2" event={"ID":"016c0a87-3f64-4eb2-ad53-925be83cdf21","Type":"ContainerDied","Data":"e972bea36f85864f0025929d86c9396271b448512db6f604534becfc4ab74c5d"}
Oct 30 01:04:27.747418 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:27.747322    4393 scope.go:122] "RemoveContainer" containerID="88e56eec75a179054b6f9cbe6461eeab23eaf2174766c8f58ac678570f367365"
Oct 30 01:04:27.760881 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:27.760857    4393 scope.go:122] "RemoveContainer" containerID="88e56eec75a179054b6f9cbe6461eeab23eaf2174766c8f58ac678570f367365"
Oct 30 01:04:27.761214 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:04:27.761190    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88e56eec75a179054b6f9cbe6461eeab23eaf2174766c8f58ac678570f367365\": container with ID starting with 88e56eec75a179054b6f9cbe6461eeab23eaf2174766c8f58ac678570f367365 not found: ID does not exist" containerID="88e56eec75a179054b6f9cbe6461eeab23eaf2174766c8f58ac678570f367365"
Oct 30 01:04:27.761293 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:27.761222    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88e56eec75a179054b6f9cbe6461eeab23eaf2174766c8f58ac678570f367365"} err="failed to get container status \"88e56eec75a179054b6f9cbe6461eeab23eaf2174766c8f58ac678570f367365\": rpc error: code = NotFound desc = could not find container \"88e56eec75a179054b6f9cbe6461eeab23eaf2174766c8f58ac678570f367365\": container with ID starting with 88e56eec75a179054b6f9cbe6461eeab23eaf2174766c8f58ac678570f367365 not found: ID does not exist"
Oct 30 01:04:27.790559 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:27.790526    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2"]
Oct 30 01:04:27.795255 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:27.795230    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8dbe73-77fcc477db-cjdb2"]
Oct 30 01:04:28.350590 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.350566    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"
Oct 30 01:04:28.533665 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.533620    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-istio-podinfo\") pod \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\" (UID: \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\") "
Oct 30 01:04:28.533870 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.533721    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-istio-token\" (UniqueName: \"kubernetes.io/projected/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-istio-token\") pod \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\" (UID: \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\") "
Oct 30 01:04:28.533950 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.533881    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-istio-envoy\") pod \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\" (UID: \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\") "
Oct 30 01:04:28.533950 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.533914    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-kube-api-access-x2bt8\" (UniqueName: \"kubernetes.io/projected/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-kube-api-access-x2bt8\") pod \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\" (UID: \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\") "
Oct 30 01:04:28.534033 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.533953    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-credential-socket\") pod \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\" (UID: \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\") "
Oct 30 01:04:28.534083 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.534030    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-workload-socket\") pod \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\" (UID: \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\") "
Oct 30 01:04:28.534083 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.534072    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-istio-data\") pod \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\" (UID: \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\") "
Oct 30 01:04:28.534190 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.534119    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-istiod-ca-cert\") pod \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\" (UID: \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\") "
Oct 30 01:04:28.534190 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.534159    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-workload-certs\") pod \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\" (UID: \"eb8b407b-2bc7-4457-814b-31ffbdffa0c2\") "
Oct 30 01:04:28.534297 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.534263    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-credential-socket" pod "eb8b407b-2bc7-4457-814b-31ffbdffa0c2" (UID: "eb8b407b-2bc7-4457-814b-31ffbdffa0c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:28.534297 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.534276    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-workload-socket" pod "eb8b407b-2bc7-4457-814b-31ffbdffa0c2" (UID: "eb8b407b-2bc7-4457-814b-31ffbdffa0c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:28.534473 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.534451    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-istio-data" pod "eb8b407b-2bc7-4457-814b-31ffbdffa0c2" (UID: "eb8b407b-2bc7-4457-814b-31ffbdffa0c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:28.534598 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.534565    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-istiod-ca-cert" pod "eb8b407b-2bc7-4457-814b-31ffbdffa0c2" (UID: "eb8b407b-2bc7-4457-814b-31ffbdffa0c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:04:28.534876 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.534852    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-workload-certs" pod "eb8b407b-2bc7-4457-814b-31ffbdffa0c2" (UID: "eb8b407b-2bc7-4457-814b-31ffbdffa0c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:28.535128 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.535109    4393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-credential-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:04:28.535170 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.535136    4393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-workload-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:04:28.535170 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.535153    4393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-istio-data\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:04:28.535234 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.535168    4393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-istiod-ca-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:04:28.535234 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.535181    4393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-workload-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:04:28.536374 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.536345    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-istio-envoy" pod "eb8b407b-2bc7-4457-814b-31ffbdffa0c2" (UID: "eb8b407b-2bc7-4457-814b-31ffbdffa0c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:28.536374 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.536352    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-kube-api-access-x2bt8" pod "eb8b407b-2bc7-4457-814b-31ffbdffa0c2" (UID: "eb8b407b-2bc7-4457-814b-31ffbdffa0c2"). InnerVolumeSpecName "kube-api-access-x2bt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:28.536637 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.536614    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-istio-podinfo" pod "eb8b407b-2bc7-4457-814b-31ffbdffa0c2" (UID: "eb8b407b-2bc7-4457-814b-31ffbdffa0c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:04:28.536740 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.536640    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-istio-token" pod "eb8b407b-2bc7-4457-814b-31ffbdffa0c2" (UID: "eb8b407b-2bc7-4457-814b-31ffbdffa0c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:28.636306 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.636276    4393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-istio-podinfo\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:04:28.636306 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.636305    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:04:28.636306 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.636314    4393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-istio-envoy\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:04:28.636519 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.636324    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2bt8\" (UniqueName: \"kubernetes.io/projected/eb8b407b-2bc7-4457-814b-31ffbdffa0c2-kube-api-access-x2bt8\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:04:28.741087 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.741058    4393 generic.go:356] "Generic (PLEG): container finished" podID="eead8403-b3bd-4a6e-bfc0-0c1785eae339" containerID="7d5e90364f94a15744f1612d3b0c3b402ad4aa16e75c3d21687784f34c0f0bad" exitCode=0
Oct 30 01:04:28.741245 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.741130    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm" event={"ID":"eead8403-b3bd-4a6e-bfc0-0c1785eae339","Type":"ContainerDied","Data":"7d5e90364f94a15744f1612d3b0c3b402ad4aa16e75c3d21687784f34c0f0bad"}
Oct 30 01:04:28.742763 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.742736    4393 generic.go:356] "Generic (PLEG): container finished" podID="eb8b407b-2bc7-4457-814b-31ffbdffa0c2" containerID="4c5c8f964a98d054cfc3f0d40790fc145fc1ca59e5bb5632b1b362282cecf9d9" exitCode=0
Oct 30 01:04:28.742945 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.742820    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8" event={"ID":"eb8b407b-2bc7-4457-814b-31ffbdffa0c2","Type":"ContainerDied","Data":"4c5c8f964a98d054cfc3f0d40790fc145fc1ca59e5bb5632b1b362282cecf9d9"}
Oct 30 01:04:28.742945 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.742830    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"
Oct 30 01:04:28.742945 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.742867    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8" event={"ID":"eb8b407b-2bc7-4457-814b-31ffbdffa0c2","Type":"ContainerDied","Data":"9db81db0bf52836e720fc46c0e04ed05053899f6e0f70e0419134a8a340a8906"}
Oct 30 01:04:28.742945 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.742893    4393 scope.go:122] "RemoveContainer" containerID="4c5c8f964a98d054cfc3f0d40790fc145fc1ca59e5bb5632b1b362282cecf9d9"
Oct 30 01:04:28.757399 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.757375    4393 scope.go:122] "RemoveContainer" containerID="4c5c8f964a98d054cfc3f0d40790fc145fc1ca59e5bb5632b1b362282cecf9d9"
Oct 30 01:04:28.757686 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:04:28.757665    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c5c8f964a98d054cfc3f0d40790fc145fc1ca59e5bb5632b1b362282cecf9d9\": container with ID starting with 4c5c8f964a98d054cfc3f0d40790fc145fc1ca59e5bb5632b1b362282cecf9d9 not found: ID does not exist" containerID="4c5c8f964a98d054cfc3f0d40790fc145fc1ca59e5bb5632b1b362282cecf9d9"
Oct 30 01:04:28.757831 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.757694    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c5c8f964a98d054cfc3f0d40790fc145fc1ca59e5bb5632b1b362282cecf9d9"} err="failed to get container status \"4c5c8f964a98d054cfc3f0d40790fc145fc1ca59e5bb5632b1b362282cecf9d9\": rpc error: code = NotFound desc = could not find container \"4c5c8f964a98d054cfc3f0d40790fc145fc1ca59e5bb5632b1b362282cecf9d9\": container with ID starting with 4c5c8f964a98d054cfc3f0d40790fc145fc1ca59e5bb5632b1b362282cecf9d9 not found: ID does not exist"
Oct 30 01:04:28.897809 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.897410    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"]
Oct 30 01:04:28.905710 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.904279    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-cefb27-openshift-default-577b649d97-tgjj8"]
Oct 30 01:04:28.924958 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.924934    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"
Oct 30 01:04:28.952362 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.952335    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eead8403-b3bd-4a6e-bfc0-0c1785eae339-istio-token\" (UniqueName: \"kubernetes.io/projected/eead8403-b3bd-4a6e-bfc0-0c1785eae339-istio-token\") pod \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\" (UID: \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\") "
Oct 30 01:04:28.952535 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.952385    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eead8403-b3bd-4a6e-bfc0-0c1785eae339-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eead8403-b3bd-4a6e-bfc0-0c1785eae339-istio-data\") pod \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\" (UID: \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\") "
Oct 30 01:04:28.952535 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.952403    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eead8403-b3bd-4a6e-bfc0-0c1785eae339-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eead8403-b3bd-4a6e-bfc0-0c1785eae339-workload-certs\") pod \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\" (UID: \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\") "
Oct 30 01:04:28.952535 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.952463    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eead8403-b3bd-4a6e-bfc0-0c1785eae339-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eead8403-b3bd-4a6e-bfc0-0c1785eae339-istiod-ca-cert\") pod \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\" (UID: \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\") "
Oct 30 01:04:28.952535 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.952482    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eead8403-b3bd-4a6e-bfc0-0c1785eae339-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eead8403-b3bd-4a6e-bfc0-0c1785eae339-credential-socket\") pod \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\" (UID: \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\") "
Oct 30 01:04:28.952769 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.952550    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eead8403-b3bd-4a6e-bfc0-0c1785eae339-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eead8403-b3bd-4a6e-bfc0-0c1785eae339-istio-podinfo\") pod \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\" (UID: \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\") "
Oct 30 01:04:28.952769 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.952609    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eead8403-b3bd-4a6e-bfc0-0c1785eae339-istio-data" pod "eead8403-b3bd-4a6e-bfc0-0c1785eae339" (UID: "eead8403-b3bd-4a6e-bfc0-0c1785eae339"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:28.952769 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.952745    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eead8403-b3bd-4a6e-bfc0-0c1785eae339-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eead8403-b3bd-4a6e-bfc0-0c1785eae339-workload-socket\") pod \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\" (UID: \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\") "
Oct 30 01:04:28.952956 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.952828    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eead8403-b3bd-4a6e-bfc0-0c1785eae339-credential-socket" pod "eead8403-b3bd-4a6e-bfc0-0c1785eae339" (UID: "eead8403-b3bd-4a6e-bfc0-0c1785eae339"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:28.952956 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.952846    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eead8403-b3bd-4a6e-bfc0-0c1785eae339-workload-certs" pod "eead8403-b3bd-4a6e-bfc0-0c1785eae339" (UID: "eead8403-b3bd-4a6e-bfc0-0c1785eae339"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:28.952956 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.952859    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eead8403-b3bd-4a6e-bfc0-0c1785eae339-kube-api-access-4tpnl\" (UniqueName: \"kubernetes.io/projected/eead8403-b3bd-4a6e-bfc0-0c1785eae339-kube-api-access-4tpnl\") pod \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\" (UID: \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\") "
Oct 30 01:04:28.953159 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.952951    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eead8403-b3bd-4a6e-bfc0-0c1785eae339-istiod-ca-cert" pod "eead8403-b3bd-4a6e-bfc0-0c1785eae339" (UID: "eead8403-b3bd-4a6e-bfc0-0c1785eae339"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:04:28.953159 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.952977    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eead8403-b3bd-4a6e-bfc0-0c1785eae339-workload-socket" pod "eead8403-b3bd-4a6e-bfc0-0c1785eae339" (UID: "eead8403-b3bd-4a6e-bfc0-0c1785eae339"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:28.953159 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.952987    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eead8403-b3bd-4a6e-bfc0-0c1785eae339-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eead8403-b3bd-4a6e-bfc0-0c1785eae339-istio-envoy\") pod \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\" (UID: \"eead8403-b3bd-4a6e-bfc0-0c1785eae339\") "
Oct 30 01:04:28.953966 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.953940    4393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eead8403-b3bd-4a6e-bfc0-0c1785eae339-istio-data\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:04:28.954078 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.953969    4393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eead8403-b3bd-4a6e-bfc0-0c1785eae339-workload-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:04:28.954078 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.953984    4393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eead8403-b3bd-4a6e-bfc0-0c1785eae339-istiod-ca-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:04:28.954078 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.953997    4393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eead8403-b3bd-4a6e-bfc0-0c1785eae339-credential-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:04:28.954078 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.954011    4393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eead8403-b3bd-4a6e-bfc0-0c1785eae339-workload-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:04:28.955166 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.955141    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eead8403-b3bd-4a6e-bfc0-0c1785eae339-istio-podinfo" pod "eead8403-b3bd-4a6e-bfc0-0c1785eae339" (UID: "eead8403-b3bd-4a6e-bfc0-0c1785eae339"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:04:28.955328 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.955311    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eead8403-b3bd-4a6e-bfc0-0c1785eae339-istio-envoy" pod "eead8403-b3bd-4a6e-bfc0-0c1785eae339" (UID: "eead8403-b3bd-4a6e-bfc0-0c1785eae339"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:28.955564 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.955539    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eead8403-b3bd-4a6e-bfc0-0c1785eae339-istio-token" pod "eead8403-b3bd-4a6e-bfc0-0c1785eae339" (UID: "eead8403-b3bd-4a6e-bfc0-0c1785eae339"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:28.955564 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:28.955554    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eead8403-b3bd-4a6e-bfc0-0c1785eae339-kube-api-access-4tpnl" pod "eead8403-b3bd-4a6e-bfc0-0c1785eae339" (UID: "eead8403-b3bd-4a6e-bfc0-0c1785eae339"). InnerVolumeSpecName "kube-api-access-4tpnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:29.055292 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:29.055257    4393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eead8403-b3bd-4a6e-bfc0-0c1785eae339-istio-podinfo\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:04:29.055292 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:29.055285    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tpnl\" (UniqueName: \"kubernetes.io/projected/eead8403-b3bd-4a6e-bfc0-0c1785eae339-kube-api-access-4tpnl\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:04:29.055292 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:29.055298    4393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eead8403-b3bd-4a6e-bfc0-0c1785eae339-istio-envoy\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:04:29.055523 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:29.055306    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eead8403-b3bd-4a6e-bfc0-0c1785eae339-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:04:29.728749 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:29.728718    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="016c0a87-3f64-4eb2-ad53-925be83cdf21" path="/var/lib/kubelet/pods/016c0a87-3f64-4eb2-ad53-925be83cdf21/volumes"
Oct 30 01:04:29.729249 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:29.729233    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb8b407b-2bc7-4457-814b-31ffbdffa0c2" path="/var/lib/kubelet/pods/eb8b407b-2bc7-4457-814b-31ffbdffa0c2/volumes"
Oct 30 01:04:29.770916 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:29.770888    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"
Oct 30 01:04:29.771107 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:29.770881    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm" event={"ID":"eead8403-b3bd-4a6e-bfc0-0c1785eae339","Type":"ContainerDied","Data":"bb443a70504bf068a78b7b6b0a41e080c6b15af549c350df987f3bf3940968e9"}
Oct 30 01:04:29.771107 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:29.771051    4393 scope.go:122] "RemoveContainer" containerID="7d5e90364f94a15744f1612d3b0c3b402ad4aa16e75c3d21687784f34c0f0bad"
Oct 30 01:04:29.838758 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:29.838727    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"]
Oct 30 01:04:29.850389 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:29.850359    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-bpwvm"]
Oct 30 01:04:31.728899 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:31.728860    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eead8403-b3bd-4a6e-bfc0-0c1785eae339" path="/var/lib/kubelet/pods/eead8403-b3bd-4a6e-bfc0-0c1785eae339/volumes"
Oct 30 01:04:34.628682 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:34.628642    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct"]
Oct 30 01:04:34.629089 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:34.628921    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct" podUID="211b2ad5-c753-4671-9be9-96b8f39aa549" containerName="maas-api" containerID="cri-o://8dd176dc9b88d239bbdffc02595fbb27fcbf07d7fb830d33e2db5a21e1424de0" gracePeriod=30
Oct 30 01:04:35.152810 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:35.146022    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct"
Oct 30 01:04:35.181805 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:35.180989    4393 generic.go:356] "Generic (PLEG): container finished" podID="211b2ad5-c753-4671-9be9-96b8f39aa549" containerID="8dd176dc9b88d239bbdffc02595fbb27fcbf07d7fb830d33e2db5a21e1424de0" exitCode=0
Oct 30 01:04:35.185339 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:35.182147    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct"
Oct 30 01:04:35.185339 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:35.182305    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct" event={"ID":"211b2ad5-c753-4671-9be9-96b8f39aa549","Type":"ContainerDied","Data":"8dd176dc9b88d239bbdffc02595fbb27fcbf07d7fb830d33e2db5a21e1424de0"}
Oct 30 01:04:35.185339 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:35.182356    4393 scope.go:122] "RemoveContainer" containerID="8dd176dc9b88d239bbdffc02595fbb27fcbf07d7fb830d33e2db5a21e1424de0"
Oct 30 01:04:35.271859 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:35.267467    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/211b2ad5-c753-4671-9be9-96b8f39aa549-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/211b2ad5-c753-4671-9be9-96b8f39aa549-maas-api-tls\") pod \"211b2ad5-c753-4671-9be9-96b8f39aa549\" (UID: \"211b2ad5-c753-4671-9be9-96b8f39aa549\") "
Oct 30 01:04:35.271859 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:35.267545    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/211b2ad5-c753-4671-9be9-96b8f39aa549-metrics-certs\" (UniqueName: \"kubernetes.io/secret/211b2ad5-c753-4671-9be9-96b8f39aa549-metrics-certs\") pod \"211b2ad5-c753-4671-9be9-96b8f39aa549\" (UID: \"211b2ad5-c753-4671-9be9-96b8f39aa549\") "
Oct 30 01:04:35.271859 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:35.267802    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/211b2ad5-c753-4671-9be9-96b8f39aa549-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/211b2ad5-c753-4671-9be9-96b8f39aa549-openshift-service-ca\") pod \"211b2ad5-c753-4671-9be9-96b8f39aa549\" (UID: \"211b2ad5-c753-4671-9be9-96b8f39aa549\") "
Oct 30 01:04:35.271859 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:35.267901    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/211b2ad5-c753-4671-9be9-96b8f39aa549-kube-api-access-tcc4j\" (UniqueName: \"kubernetes.io/projected/211b2ad5-c753-4671-9be9-96b8f39aa549-kube-api-access-tcc4j\") pod \"211b2ad5-c753-4671-9be9-96b8f39aa549\" (UID: \"211b2ad5-c753-4671-9be9-96b8f39aa549\") "
Oct 30 01:04:35.287811 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:35.276255    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/211b2ad5-c753-4671-9be9-96b8f39aa549-openshift-service-ca" pod "211b2ad5-c753-4671-9be9-96b8f39aa549" (UID: "211b2ad5-c753-4671-9be9-96b8f39aa549"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:04:35.321431 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:35.321369    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/211b2ad5-c753-4671-9be9-96b8f39aa549-kube-api-access-tcc4j" pod "211b2ad5-c753-4671-9be9-96b8f39aa549" (UID: "211b2ad5-c753-4671-9be9-96b8f39aa549"). InnerVolumeSpecName "kube-api-access-tcc4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:35.322976 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:35.322929    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/211b2ad5-c753-4671-9be9-96b8f39aa549-maas-api-tls" pod "211b2ad5-c753-4671-9be9-96b8f39aa549" (UID: "211b2ad5-c753-4671-9be9-96b8f39aa549"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:04:35.346121 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:35.334823    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s"]
Oct 30 01:04:35.346121 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:35.335452    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/211b2ad5-c753-4671-9be9-96b8f39aa549-metrics-certs" pod "211b2ad5-c753-4671-9be9-96b8f39aa549" (UID: "211b2ad5-c753-4671-9be9-96b8f39aa549"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:04:35.370288 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:35.370259    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/211b2ad5-c753-4671-9be9-96b8f39aa549-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:04:35.370467 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:35.370457    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcc4j\" (UniqueName: \"kubernetes.io/projected/211b2ad5-c753-4671-9be9-96b8f39aa549-kube-api-access-tcc4j\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:04:35.370528 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:35.370519    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/211b2ad5-c753-4671-9be9-96b8f39aa549-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:04:35.370591 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:35.370583    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/211b2ad5-c753-4671-9be9-96b8f39aa549-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:04:35.601885 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:35.601849    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct"]
Oct 30 01:04:35.606230 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:35.606202    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-03d892-f7f75b95b-4w7ct"]
Oct 30 01:04:35.728197 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:35.728168    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="211b2ad5-c753-4671-9be9-96b8f39aa549" path="/var/lib/kubelet/pods/211b2ad5-c753-4671-9be9-96b8f39aa549/volumes"
Oct 30 01:04:46.225944 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.225909    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"]
Oct 30 01:04:46.226930 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.226904    4393 cpu_manager.go:414] "containerMap: removing container" podUID="016c0a87-3f64-4eb2-ad53-925be83cdf21" containerName="maas-api"
Oct 30 01:04:46.226930 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.226928    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="016c0a87-3f64-4eb2-ad53-925be83cdf21" containerName="maas-api"
Oct 30 01:04:46.227128 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.226991    4393 cpu_manager.go:414] "containerMap: removing container" podUID="eead8403-b3bd-4a6e-bfc0-0c1785eae339" containerName="istio-proxy"
Oct 30 01:04:46.227128 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.226999    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eead8403-b3bd-4a6e-bfc0-0c1785eae339" containerName="istio-proxy"
Oct 30 01:04:46.227128 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.227013    4393 cpu_manager.go:414] "containerMap: removing container" podUID="b7357139-9713-42ac-bdbe-c9093760765b" containerName="maas-api"
Oct 30 01:04:46.227128 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.227020    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7357139-9713-42ac-bdbe-c9093760765b" containerName="maas-api"
Oct 30 01:04:46.227128 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.227043    4393 cpu_manager.go:414] "containerMap: removing container" podUID="211b2ad5-c753-4671-9be9-96b8f39aa549" containerName="maas-api"
Oct 30 01:04:46.227128 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.227050    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="211b2ad5-c753-4671-9be9-96b8f39aa549" containerName="maas-api"
Oct 30 01:04:46.227128 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.227069    4393 cpu_manager.go:414] "containerMap: removing container" podUID="eb8b407b-2bc7-4457-814b-31ffbdffa0c2" containerName="istio-proxy"
Oct 30 01:04:46.227128 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.227076    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb8b407b-2bc7-4457-814b-31ffbdffa0c2" containerName="istio-proxy"
Oct 30 01:04:46.227513 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.227273    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="016c0a87-3f64-4eb2-ad53-925be83cdf21" containerName="maas-api"
Oct 30 01:04:46.227513 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.227296    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="eead8403-b3bd-4a6e-bfc0-0c1785eae339" containerName="istio-proxy"
Oct 30 01:04:46.227513 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.227307    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7357139-9713-42ac-bdbe-c9093760765b" containerName="maas-api"
Oct 30 01:04:46.227513 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.227319    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="211b2ad5-c753-4671-9be9-96b8f39aa549" containerName="maas-api"
Oct 30 01:04:46.227513 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.227332    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb8b407b-2bc7-4457-814b-31ffbdffa0c2" containerName="istio-proxy"
Oct 30 01:04:46.230371 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.230348    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"
Oct 30 01:04:46.431914 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.431866    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-istiod-ca-cert\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn\" (UID: \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"
Oct 30 01:04:46.432144 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.432026    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-istio-podinfo\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn\" (UID: \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"
Oct 30 01:04:46.432144 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.432097    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-istio-token\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn\" (UID: \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"
Oct 30 01:04:46.432267 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.432150    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-credential-socket\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn\" (UID: \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"
Oct 30 01:04:46.432267 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.432211    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-istio-envoy\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn\" (UID: \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"
Oct 30 01:04:46.432369 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.432332    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-workload-socket\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn\" (UID: \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"
Oct 30 01:04:46.432421 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.432380    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzhj6\" (UniqueName: \"kubernetes.io/projected/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-kube-api-access-bzhj6\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn\" (UID: \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"
Oct 30 01:04:46.432572 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.432545    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-workload-certs\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn\" (UID: \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"
Oct 30 01:04:46.432643 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.432593    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-istio-data\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn\" (UID: \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"
Oct 30 01:04:46.447820 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.433503    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ef1084-openshift-default-dockercfg-57vpk\""
Oct 30 01:04:46.456165 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.456127    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"]
Oct 30 01:04:46.534113 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.534023    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-workload-certs\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn\" (UID: \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"
Oct 30 01:04:46.534113 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.534064    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-istio-data\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn\" (UID: \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"
Oct 30 01:04:46.534385 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.534124    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-istiod-ca-cert\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn\" (UID: \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"
Oct 30 01:04:46.534385 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.534188    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-istio-podinfo\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn\" (UID: \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"
Oct 30 01:04:46.534385 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.534221    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-istio-token\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn\" (UID: \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"
Oct 30 01:04:46.534385 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.534246    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-credential-socket\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn\" (UID: \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"
Oct 30 01:04:46.534385 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.534275    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-istio-envoy\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn\" (UID: \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"
Oct 30 01:04:46.534385 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.534339    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-workload-socket\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn\" (UID: \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"
Oct 30 01:04:46.534385 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.534363    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzhj6\" (UniqueName: \"kubernetes.io/projected/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-kube-api-access-bzhj6\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn\" (UID: \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"
Oct 30 01:04:46.534799 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.534537    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-istio-data\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn\" (UID: \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"
Oct 30 01:04:46.534799 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.534565    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-workload-certs\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn\" (UID: \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"
Oct 30 01:04:46.534925 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.534852    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-credential-socket\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn\" (UID: \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"
Oct 30 01:04:46.534974 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.534923    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-workload-socket\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn\" (UID: \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"
Oct 30 01:04:46.535276 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.535258    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-istiod-ca-cert\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn\" (UID: \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"
Oct 30 01:04:46.536931 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.536909    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-istio-envoy\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn\" (UID: \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"
Oct 30 01:04:46.537468 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.537446    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-istio-podinfo\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn\" (UID: \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"
Oct 30 01:04:46.560439 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.560409    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-istio-token\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn\" (UID: \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"
Oct 30 01:04:46.560575 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.560502    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzhj6\" (UniqueName: \"kubernetes.io/projected/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-kube-api-access-bzhj6\") pod \"e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn\" (UID: \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\") " pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"
Oct 30 01:04:46.734264 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.734230    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s"]
Oct 30 01:04:46.874806 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:46.873212    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"
Oct 30 01:04:47.255812 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.255700    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"]
Oct 30 01:04:47.268041 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.268011    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"]
Oct 30 01:04:47.268194 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.268113    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"
Oct 30 01:04:47.296927 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.296896    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-48df96-openshift-default-dockercfg-h6qzs\""
Oct 30 01:04:47.310959 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:04:47.310932    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f8bf91a_d2e7_4c6e_90c4_e8d794029f21.slice/crio-474be8e4c93406fd789f049d619cb59a80239a1abb881b71cbb09d9242060ae6 WatchSource:0}: Error finding container 474be8e4c93406fd789f049d619cb59a80239a1abb881b71cbb09d9242060ae6: Status 404 returned error can't find the container with id 474be8e4c93406fd789f049d619cb59a80239a1abb881b71cbb09d9242060ae6
Oct 30 01:04:47.311514 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.311490    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"]
Oct 30 01:04:47.313633 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.313597    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:04:47.313757 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.313675    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:04:47.313757 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.313719    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:04:47.350564 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.350534    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2535b52d-1db6-4f2b-9912-4d9316aaa69d-istio-envoy\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh\" (UID: \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"
Oct 30 01:04:47.350683 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.350571    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2535b52d-1db6-4f2b-9912-4d9316aaa69d-istiod-ca-cert\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh\" (UID: \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"
Oct 30 01:04:47.350729 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.350713    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2535b52d-1db6-4f2b-9912-4d9316aaa69d-istio-token\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh\" (UID: \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"
Oct 30 01:04:47.350884 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.350864    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2535b52d-1db6-4f2b-9912-4d9316aaa69d-credential-socket\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh\" (UID: \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"
Oct 30 01:04:47.350933 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.350906    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2535b52d-1db6-4f2b-9912-4d9316aaa69d-workload-certs\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh\" (UID: \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"
Oct 30 01:04:47.350933 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.350927    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbbxc\" (UniqueName: \"kubernetes.io/projected/2535b52d-1db6-4f2b-9912-4d9316aaa69d-kube-api-access-qbbxc\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh\" (UID: \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"
Oct 30 01:04:47.351008 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.350987    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2535b52d-1db6-4f2b-9912-4d9316aaa69d-workload-socket\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh\" (UID: \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"
Oct 30 01:04:47.351049 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.351035    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2535b52d-1db6-4f2b-9912-4d9316aaa69d-istio-data\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh\" (UID: \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"
Oct 30 01:04:47.351187 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.351161    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2535b52d-1db6-4f2b-9912-4d9316aaa69d-istio-podinfo\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh\" (UID: \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"
Oct 30 01:04:47.452817 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.452760    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2535b52d-1db6-4f2b-9912-4d9316aaa69d-istio-envoy\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh\" (UID: \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"
Oct 30 01:04:47.453017 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.452820    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2535b52d-1db6-4f2b-9912-4d9316aaa69d-istiod-ca-cert\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh\" (UID: \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"
Oct 30 01:04:47.453017 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.452956    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2535b52d-1db6-4f2b-9912-4d9316aaa69d-istio-token\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh\" (UID: \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"
Oct 30 01:04:47.453148 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.453065    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2535b52d-1db6-4f2b-9912-4d9316aaa69d-credential-socket\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh\" (UID: \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"
Oct 30 01:04:47.453148 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.453112    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2535b52d-1db6-4f2b-9912-4d9316aaa69d-workload-certs\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh\" (UID: \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"
Oct 30 01:04:47.453306 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.453227    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbbxc\" (UniqueName: \"kubernetes.io/projected/2535b52d-1db6-4f2b-9912-4d9316aaa69d-kube-api-access-qbbxc\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh\" (UID: \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"
Oct 30 01:04:47.453306 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.453277    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2535b52d-1db6-4f2b-9912-4d9316aaa69d-workload-socket\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh\" (UID: \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"
Oct 30 01:04:47.453453 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.453342    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2535b52d-1db6-4f2b-9912-4d9316aaa69d-istio-data\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh\" (UID: \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"
Oct 30 01:04:47.453453 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.453434    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2535b52d-1db6-4f2b-9912-4d9316aaa69d-credential-socket\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh\" (UID: \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"
Oct 30 01:04:47.453453 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.453445    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2535b52d-1db6-4f2b-9912-4d9316aaa69d-workload-certs\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh\" (UID: \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"
Oct 30 01:04:47.453612 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.453449    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2535b52d-1db6-4f2b-9912-4d9316aaa69d-istio-podinfo\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh\" (UID: \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"
Oct 30 01:04:47.453612 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.453583    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2535b52d-1db6-4f2b-9912-4d9316aaa69d-istiod-ca-cert\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh\" (UID: \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"
Oct 30 01:04:47.453753 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.453731    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2535b52d-1db6-4f2b-9912-4d9316aaa69d-istio-data\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh\" (UID: \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"
Oct 30 01:04:47.453827 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.453755    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2535b52d-1db6-4f2b-9912-4d9316aaa69d-workload-socket\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh\" (UID: \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"
Oct 30 01:04:47.455523 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.455497    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2535b52d-1db6-4f2b-9912-4d9316aaa69d-istio-envoy\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh\" (UID: \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"
Oct 30 01:04:47.456293 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.456267    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2535b52d-1db6-4f2b-9912-4d9316aaa69d-istio-podinfo\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh\" (UID: \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"
Oct 30 01:04:47.476423 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.476387    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2535b52d-1db6-4f2b-9912-4d9316aaa69d-istio-token\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh\" (UID: \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"
Oct 30 01:04:47.476666 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.476649    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbbxc\" (UniqueName: \"kubernetes.io/projected/2535b52d-1db6-4f2b-9912-4d9316aaa69d-kube-api-access-qbbxc\") pod \"e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh\" (UID: \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\") " pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"
Oct 30 01:04:47.585846 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.585751    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"
Oct 30 01:04:47.923400 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:04:47.923366    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2535b52d_1db6_4f2b_9912_4d9316aaa69d.slice/crio-7ce993e3367534b37827c5996c9db14d6f50b6b239f7207d10836228bbaa8d85 WatchSource:0}: Error finding container 7ce993e3367534b37827c5996c9db14d6f50b6b239f7207d10836228bbaa8d85: Status 404 returned error can't find the container with id 7ce993e3367534b37827c5996c9db14d6f50b6b239f7207d10836228bbaa8d85
Oct 30 01:04:47.924004 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.923977    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"]
Oct 30 01:04:47.926758 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.926724    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:04:47.926897 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.926840    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:04:47.926897 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:47.926882    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:04:48.344313 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:48.343986    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn" event={"ID":"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21","Type":"ContainerStarted","Data":"2530e15d554970bf633648513bbe32e408b1ced3ce4390b170b8593257b80e51"}
Oct 30 01:04:48.344313 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:48.344042    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn" event={"ID":"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21","Type":"ContainerStarted","Data":"474be8e4c93406fd789f049d619cb59a80239a1abb881b71cbb09d9242060ae6"}
Oct 30 01:04:48.347081 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:48.347055    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh" event={"ID":"2535b52d-1db6-4f2b-9912-4d9316aaa69d","Type":"ContainerStarted","Data":"e0e85a64e815237d97347aa41b31af9d929c4631a949e711b9e20a9acde2dbeb"}
Oct 30 01:04:48.347179 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:48.347100    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh" event={"ID":"2535b52d-1db6-4f2b-9912-4d9316aaa69d","Type":"ContainerStarted","Data":"7ce993e3367534b37827c5996c9db14d6f50b6b239f7207d10836228bbaa8d85"}
Oct 30 01:04:48.436140 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:48.436075    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn" podStartSLOduration=2.436057993 podStartE2EDuration="2.436057993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:48.422975585 +0000 UTC m=+1978.810651975" watchObservedRunningTime="2026-10-30 01:04:48.436057993 +0000 UTC m=+1978.823734381"
Oct 30 01:04:48.477348 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:48.477282    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh" podStartSLOduration=1.477261465 podStartE2EDuration="1.477261465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:04:48.468816901 +0000 UTC m=+1978.856493300" watchObservedRunningTime="2026-10-30 01:04:48.477261465 +0000 UTC m=+1978.864937856"
Oct 30 01:04:48.586548 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:48.586519    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"
Oct 30 01:04:48.892552 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:48.892512    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"
Oct 30 01:04:48.895091 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:48.895056    4393 patch_prober.go:29] interesting pod/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 01:04:48.895217 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:48.895111    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn" podUID="0f8bf91a-d2e7-4c6e-90c4-e8d794029f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 01:04:49.588274 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:49.588239    4393 patch_prober.go:29] interesting pod/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 01:04:49.588871 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:49.588297    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh" podUID="2535b52d-1db6-4f2b-9912-4d9316aaa69d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 01:04:49.892937 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:49.892905    4393 patch_prober.go:29] interesting pod/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 01:04:49.892937 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:49.892951    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn" podUID="0f8bf91a-d2e7-4c6e-90c4-e8d794029f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 01:04:50.408804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:50.390990    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s" podUID="27318ea7-6cda-457e-adc9-33467978bbbc" containerName="main" containerID="cri-o://50ee28355f8f93524eb36392ec71062f06ac5a901b4c615e7fe01ceb7aa64176" gracePeriod=45
Oct 30 01:04:50.773859 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:50.773084    4393 patch_prober.go:29] interesting pod/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 01:04:50.773859 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:50.773158    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh" podUID="2535b52d-1db6-4f2b-9912-4d9316aaa69d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 01:04:50.884850 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:50.883919    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s" podUID="27318ea7-6cda-457e-adc9-33467978bbbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.144:8000/ready\": dial tcp 10.133.0.144:8000: connect: connection refused"
Oct 30 01:04:50.893049 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:50.893027    4393 patch_prober.go:29] interesting pod/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 01:04:50.893129 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:50.893072    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn" podUID="0f8bf91a-d2e7-4c6e-90c4-e8d794029f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 01:04:51.281959 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:51.281929    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s"
Oct 30 01:04:51.296329 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:51.296299    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27318ea7-6cda-457e-adc9-33467978bbbc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27318ea7-6cda-457e-adc9-33467978bbbc-tmp-dir\") pod \"27318ea7-6cda-457e-adc9-33467978bbbc\" (UID: \"27318ea7-6cda-457e-adc9-33467978bbbc\") "
Oct 30 01:04:51.296492 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:51.296367    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27318ea7-6cda-457e-adc9-33467978bbbc-kube-api-access-w7jtv\" (UniqueName: \"kubernetes.io/projected/27318ea7-6cda-457e-adc9-33467978bbbc-kube-api-access-w7jtv\") pod \"27318ea7-6cda-457e-adc9-33467978bbbc\" (UID: \"27318ea7-6cda-457e-adc9-33467978bbbc\") "
Oct 30 01:04:51.296492 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:51.296395    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27318ea7-6cda-457e-adc9-33467978bbbc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/27318ea7-6cda-457e-adc9-33467978bbbc-dshm\") pod \"27318ea7-6cda-457e-adc9-33467978bbbc\" (UID: \"27318ea7-6cda-457e-adc9-33467978bbbc\") "
Oct 30 01:04:51.296492 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:51.296441    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27318ea7-6cda-457e-adc9-33467978bbbc-home\" (UniqueName: \"kubernetes.io/empty-dir/27318ea7-6cda-457e-adc9-33467978bbbc-home\") pod \"27318ea7-6cda-457e-adc9-33467978bbbc\" (UID: \"27318ea7-6cda-457e-adc9-33467978bbbc\") "
Oct 30 01:04:51.296689 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:51.296547    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27318ea7-6cda-457e-adc9-33467978bbbc-tls-certs\" (UniqueName: \"kubernetes.io/secret/27318ea7-6cda-457e-adc9-33467978bbbc-tls-certs\") pod \"27318ea7-6cda-457e-adc9-33467978bbbc\" (UID: \"27318ea7-6cda-457e-adc9-33467978bbbc\") "
Oct 30 01:04:51.296689 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:51.296600    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27318ea7-6cda-457e-adc9-33467978bbbc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27318ea7-6cda-457e-adc9-33467978bbbc-model-cache\") pod \"27318ea7-6cda-457e-adc9-33467978bbbc\" (UID: \"27318ea7-6cda-457e-adc9-33467978bbbc\") "
Oct 30 01:04:51.297281 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:51.297240    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27318ea7-6cda-457e-adc9-33467978bbbc-home" pod "27318ea7-6cda-457e-adc9-33467978bbbc" (UID: "27318ea7-6cda-457e-adc9-33467978bbbc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:51.297405 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:51.297298    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27318ea7-6cda-457e-adc9-33467978bbbc-model-cache" pod "27318ea7-6cda-457e-adc9-33467978bbbc" (UID: "27318ea7-6cda-457e-adc9-33467978bbbc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:51.297719 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:51.297635    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27318ea7-6cda-457e-adc9-33467978bbbc-tmp-dir" pod "27318ea7-6cda-457e-adc9-33467978bbbc" (UID: "27318ea7-6cda-457e-adc9-33467978bbbc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:51.299631 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:51.299593    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27318ea7-6cda-457e-adc9-33467978bbbc-tls-certs" pod "27318ea7-6cda-457e-adc9-33467978bbbc" (UID: "27318ea7-6cda-457e-adc9-33467978bbbc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:04:51.299726 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:51.299634    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27318ea7-6cda-457e-adc9-33467978bbbc-kube-api-access-w7jtv" pod "27318ea7-6cda-457e-adc9-33467978bbbc" (UID: "27318ea7-6cda-457e-adc9-33467978bbbc"). InnerVolumeSpecName "kube-api-access-w7jtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:04:51.301488 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:51.301464    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27318ea7-6cda-457e-adc9-33467978bbbc-dshm" pod "27318ea7-6cda-457e-adc9-33467978bbbc" (UID: "27318ea7-6cda-457e-adc9-33467978bbbc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:04:51.398684 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:51.398654    4393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27318ea7-6cda-457e-adc9-33467978bbbc-model-cache\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:04:51.398684 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:51.398682    4393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27318ea7-6cda-457e-adc9-33467978bbbc-tmp-dir\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:04:51.398684 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:51.398691    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7jtv\" (UniqueName: \"kubernetes.io/projected/27318ea7-6cda-457e-adc9-33467978bbbc-kube-api-access-w7jtv\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:04:51.398943 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:51.398701    4393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27318ea7-6cda-457e-adc9-33467978bbbc-dshm\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:04:51.398943 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:51.398709    4393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27318ea7-6cda-457e-adc9-33467978bbbc-home\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:04:51.398943 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:51.398716    4393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27318ea7-6cda-457e-adc9-33467978bbbc-tls-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:04:51.586350 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:51.586316    4393 patch_prober.go:29] interesting pod/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 01:04:51.586533 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:51.586371    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh" podUID="2535b52d-1db6-4f2b-9912-4d9316aaa69d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 01:04:51.587161 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:51.587131    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb"]
Oct 30 01:04:51.912806 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:51.893365    4393 patch_prober.go:29] interesting pod/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 01:04:51.912806 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:51.893446    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn" podUID="0f8bf91a-d2e7-4c6e-90c4-e8d794029f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 01:04:51.967735 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:51.967688    4393 generic.go:356] "Generic (PLEG): container finished" podID="27318ea7-6cda-457e-adc9-33467978bbbc" containerID="50ee28355f8f93524eb36392ec71062f06ac5a901b4c615e7fe01ceb7aa64176" exitCode=0
Oct 30 01:04:51.967929 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:51.967893    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s" event={"ID":"27318ea7-6cda-457e-adc9-33467978bbbc","Type":"ContainerDied","Data":"50ee28355f8f93524eb36392ec71062f06ac5a901b4c615e7fe01ceb7aa64176"}
Oct 30 01:04:51.968015 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:51.967929    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s" event={"ID":"27318ea7-6cda-457e-adc9-33467978bbbc","Type":"ContainerDied","Data":"aa443bb6ff04b73905981062e36e5880346d2020c74fc5c9166f36d2cb0ab294"}
Oct 30 01:04:51.968015 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:51.967954    4393 scope.go:122] "RemoveContainer" containerID="50ee28355f8f93524eb36392ec71062f06ac5a901b4c615e7fe01ceb7aa64176"
Oct 30 01:04:51.968126 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:51.968112    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s"
Oct 30 01:04:51.988771 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:51.988749    4393 scope.go:122] "RemoveContainer" containerID="50ee28355f8f93524eb36392ec71062f06ac5a901b4c615e7fe01ceb7aa64176"
Oct 30 01:04:51.989196 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:04:51.989172    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50ee28355f8f93524eb36392ec71062f06ac5a901b4c615e7fe01ceb7aa64176\": container with ID starting with 50ee28355f8f93524eb36392ec71062f06ac5a901b4c615e7fe01ceb7aa64176 not found: ID does not exist" containerID="50ee28355f8f93524eb36392ec71062f06ac5a901b4c615e7fe01ceb7aa64176"
Oct 30 01:04:51.989309 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:51.989202    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50ee28355f8f93524eb36392ec71062f06ac5a901b4c615e7fe01ceb7aa64176"} err="failed to get container status \"50ee28355f8f93524eb36392ec71062f06ac5a901b4c615e7fe01ceb7aa64176\": rpc error: code = NotFound desc = could not find container \"50ee28355f8f93524eb36392ec71062f06ac5a901b4c615e7fe01ceb7aa64176\": container with ID starting with 50ee28355f8f93524eb36392ec71062f06ac5a901b4c615e7fe01ceb7aa64176 not found: ID does not exist"
Oct 30 01:04:52.063018 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:52.062974    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s"]
Oct 30 01:04:52.068179 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:52.068154    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-8d2a70/test-model-8d2a70-kserve-557bb6c9bc-h766s"]
Oct 30 01:04:52.586145 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:52.586109    4393 patch_prober.go:29] interesting pod/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 01:04:52.586334 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:52.586177    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh" podUID="2535b52d-1db6-4f2b-9912-4d9316aaa69d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 01:04:52.893429 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:52.893397    4393 patch_prober.go:29] interesting pod/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 01:04:52.893598 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:52.893449    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn" podUID="0f8bf91a-d2e7-4c6e-90c4-e8d794029f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 01:04:53.586445 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:53.586403    4393 patch_prober.go:29] interesting pod/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 01:04:53.586976 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:53.586466    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh" podUID="2535b52d-1db6-4f2b-9912-4d9316aaa69d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 01:04:53.763267 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:53.763176    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27318ea7-6cda-457e-adc9-33467978bbbc" path="/var/lib/kubelet/pods/27318ea7-6cda-457e-adc9-33467978bbbc/volumes"
Oct 30 01:04:53.893034 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:53.893004    4393 patch_prober.go:29] interesting pod/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 01:04:53.893204 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:53.893051    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn" podUID="0f8bf91a-d2e7-4c6e-90c4-e8d794029f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 01:04:54.587025 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:54.586977    4393 patch_prober.go:29] interesting pod/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Oct 30 01:04:54.587673 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:54.587034    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh" podUID="2535b52d-1db6-4f2b-9912-4d9316aaa69d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Oct 30 01:04:54.898741 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:54.898708    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"
Oct 30 01:04:54.899255 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:54.899230    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"
Oct 30 01:04:54.900096 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:54.900078    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"
Oct 30 01:04:55.592016 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:55.591984    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"
Oct 30 01:04:55.592531 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:55.592484    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"
Oct 30 01:04:55.593149 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:55.593127    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"
Oct 30 01:04:56.513109 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:04:56.513067    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:04:58.787655 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:58.787622    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"]
Oct 30 01:04:58.788415 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:58.788400    4393 cpu_manager.go:414] "containerMap: removing container" podUID="27318ea7-6cda-457e-adc9-33467978bbbc" containerName="main"
Oct 30 01:04:58.788460 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:58.788416    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27318ea7-6cda-457e-adc9-33467978bbbc" containerName="main"
Oct 30 01:04:58.788581 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:58.788572    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="27318ea7-6cda-457e-adc9-33467978bbbc" containerName="main"
Oct 30 01:04:58.792289 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:58.792270    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"
Oct 30 01:04:58.819060 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:58.819027    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-cce513-openshift-default-dockercfg-7qp6x\""
Oct 30 01:04:58.827113 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:58.827081    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"]
Oct 30 01:04:58.885217 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:58.885185    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/123708b7-bba7-4cc1-a2c0-9f3887a742f1-istiod-ca-cert\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc\" (UID: \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"
Oct 30 01:04:58.885217 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:58.885232    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/123708b7-bba7-4cc1-a2c0-9f3887a742f1-workload-socket\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc\" (UID: \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"
Oct 30 01:04:58.885491 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:58.885401    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/123708b7-bba7-4cc1-a2c0-9f3887a742f1-credential-socket\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc\" (UID: \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"
Oct 30 01:04:58.885491 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:58.885420    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/123708b7-bba7-4cc1-a2c0-9f3887a742f1-istio-envoy\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc\" (UID: \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"
Oct 30 01:04:58.885491 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:58.885475    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/123708b7-bba7-4cc1-a2c0-9f3887a742f1-istio-token\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc\" (UID: \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"
Oct 30 01:04:58.885656 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:58.885522    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/123708b7-bba7-4cc1-a2c0-9f3887a742f1-istio-data\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc\" (UID: \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"
Oct 30 01:04:58.885719 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:58.885666    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/123708b7-bba7-4cc1-a2c0-9f3887a742f1-workload-certs\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc\" (UID: \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"
Oct 30 01:04:58.885719 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:58.885694    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/123708b7-bba7-4cc1-a2c0-9f3887a742f1-istio-podinfo\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc\" (UID: \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"
Oct 30 01:04:58.885888 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:58.885859    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmf82\" (UniqueName: \"kubernetes.io/projected/123708b7-bba7-4cc1-a2c0-9f3887a742f1-kube-api-access-cmf82\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc\" (UID: \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"
Oct 30 01:04:58.987135 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:58.987098    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/123708b7-bba7-4cc1-a2c0-9f3887a742f1-credential-socket\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc\" (UID: \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"
Oct 30 01:04:58.987135 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:58.987138    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/123708b7-bba7-4cc1-a2c0-9f3887a742f1-istio-envoy\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc\" (UID: \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"
Oct 30 01:04:58.987398 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:58.987197    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/123708b7-bba7-4cc1-a2c0-9f3887a742f1-istio-token\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc\" (UID: \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"
Oct 30 01:04:58.987398 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:58.987225    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/123708b7-bba7-4cc1-a2c0-9f3887a742f1-istio-data\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc\" (UID: \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"
Oct 30 01:04:58.987398 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:58.987386    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/123708b7-bba7-4cc1-a2c0-9f3887a742f1-workload-certs\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc\" (UID: \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"
Oct 30 01:04:58.987567 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:58.987419    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/123708b7-bba7-4cc1-a2c0-9f3887a742f1-istio-podinfo\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc\" (UID: \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"
Oct 30 01:04:58.987567 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:58.987555    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmf82\" (UniqueName: \"kubernetes.io/projected/123708b7-bba7-4cc1-a2c0-9f3887a742f1-kube-api-access-cmf82\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc\" (UID: \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"
Oct 30 01:04:58.987679 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:58.987570    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/123708b7-bba7-4cc1-a2c0-9f3887a742f1-credential-socket\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc\" (UID: \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"
Oct 30 01:04:58.987679 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:58.987588    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/123708b7-bba7-4cc1-a2c0-9f3887a742f1-istiod-ca-cert\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc\" (UID: \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"
Oct 30 01:04:58.987679 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:58.987623    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/123708b7-bba7-4cc1-a2c0-9f3887a742f1-workload-socket\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc\" (UID: \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"
Oct 30 01:04:58.987899 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:58.987868    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/123708b7-bba7-4cc1-a2c0-9f3887a742f1-workload-certs\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc\" (UID: \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"
Oct 30 01:04:58.988058 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:58.988037    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/123708b7-bba7-4cc1-a2c0-9f3887a742f1-workload-socket\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc\" (UID: \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"
Oct 30 01:04:58.988123 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:58.988037    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/123708b7-bba7-4cc1-a2c0-9f3887a742f1-istio-data\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc\" (UID: \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"
Oct 30 01:04:58.988609 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:58.988585    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/123708b7-bba7-4cc1-a2c0-9f3887a742f1-istiod-ca-cert\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc\" (UID: \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"
Oct 30 01:04:58.989869 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:58.989844    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/123708b7-bba7-4cc1-a2c0-9f3887a742f1-istio-envoy\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc\" (UID: \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"
Oct 30 01:04:58.990165 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:58.990146    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/123708b7-bba7-4cc1-a2c0-9f3887a742f1-istio-podinfo\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc\" (UID: \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"
Oct 30 01:04:59.009603 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:59.009568    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/123708b7-bba7-4cc1-a2c0-9f3887a742f1-istio-token\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc\" (UID: \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"
Oct 30 01:04:59.009836 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:59.009815    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmf82\" (UniqueName: \"kubernetes.io/projected/123708b7-bba7-4cc1-a2c0-9f3887a742f1-kube-api-access-cmf82\") pod \"e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc\" (UID: \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\") " pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"
Oct 30 01:04:59.108315 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:59.108230    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"
Oct 30 01:04:59.439976 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:04:59.439942    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod123708b7_bba7_4cc1_a2c0_9f3887a742f1.slice/crio-c60ce85eb2210cb6634f9d1ba5fe4dc1b5572ffd6f78559e7ecf4f2e4827fe81 WatchSource:0}: Error finding container c60ce85eb2210cb6634f9d1ba5fe4dc1b5572ffd6f78559e7ecf4f2e4827fe81: Status 404 returned error can't find the container with id c60ce85eb2210cb6634f9d1ba5fe4dc1b5572ffd6f78559e7ecf4f2e4827fe81
Oct 30 01:04:59.440444 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:59.440417    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"]
Oct 30 01:04:59.443626 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:59.443588    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:04:59.443742 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:59.443659    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:04:59.443742 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:04:59.443692    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:05:00.051532 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.051490    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc" event={"ID":"123708b7-bba7-4cc1-a2c0-9f3887a742f1","Type":"ContainerStarted","Data":"2f2fdbbc17929bf3073d94f41ca2218364bf82daa842cdaf50abe3c6e0b1b13d"}
Oct 30 01:05:00.052045 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.051560    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc" event={"ID":"123708b7-bba7-4cc1-a2c0-9f3887a742f1","Type":"ContainerStarted","Data":"c60ce85eb2210cb6634f9d1ba5fe4dc1b5572ffd6f78559e7ecf4f2e4827fe81"}
Oct 30 01:05:00.052045 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.051961    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"]
Oct 30 01:05:00.056817 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.056757    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"
Oct 30 01:05:00.069530 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.069507    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"]
Oct 30 01:05:00.074868 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.074844    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b1f6ed-openshift-default-dockercfg-wjgfj\""
Oct 30 01:05:00.098473 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.098416    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc" podStartSLOduration=2.098400684 podStartE2EDuration="2.098400684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:00.094924676 +0000 UTC m=+1990.482601067" watchObservedRunningTime="2026-10-30 01:05:00.098400684 +0000 UTC m=+1990.486077074"
Oct 30 01:05:00.114362 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.114326    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"
Oct 30 01:05:00.129533 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.129491    4393 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 01:05:00.129727 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.129561    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc" podUID="123708b7-bba7-4cc1-a2c0-9f3887a742f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 01:05:00.229878 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.224749    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f7dc119-7915-473a-b425-b9fdd217a8e8-workload-certs\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d\" (UID: \"5f7dc119-7915-473a-b425-b9fdd217a8e8\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"
Oct 30 01:05:00.229878 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.224855    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzfph\" (UniqueName: \"kubernetes.io/projected/5f7dc119-7915-473a-b425-b9fdd217a8e8-kube-api-access-fzfph\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d\" (UID: \"5f7dc119-7915-473a-b425-b9fdd217a8e8\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"
Oct 30 01:05:00.229878 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.224889    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f7dc119-7915-473a-b425-b9fdd217a8e8-istio-data\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d\" (UID: \"5f7dc119-7915-473a-b425-b9fdd217a8e8\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"
Oct 30 01:05:00.229878 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.224938    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f7dc119-7915-473a-b425-b9fdd217a8e8-istio-token\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d\" (UID: \"5f7dc119-7915-473a-b425-b9fdd217a8e8\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"
Oct 30 01:05:00.229878 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.225010    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f7dc119-7915-473a-b425-b9fdd217a8e8-istiod-ca-cert\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d\" (UID: \"5f7dc119-7915-473a-b425-b9fdd217a8e8\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"
Oct 30 01:05:00.229878 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.225069    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7dc119-7915-473a-b425-b9fdd217a8e8-credential-socket\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d\" (UID: \"5f7dc119-7915-473a-b425-b9fdd217a8e8\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"
Oct 30 01:05:00.229878 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.225108    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f7dc119-7915-473a-b425-b9fdd217a8e8-istio-podinfo\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d\" (UID: \"5f7dc119-7915-473a-b425-b9fdd217a8e8\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"
Oct 30 01:05:00.229878 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.225185    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f7dc119-7915-473a-b425-b9fdd217a8e8-istio-envoy\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d\" (UID: \"5f7dc119-7915-473a-b425-b9fdd217a8e8\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"
Oct 30 01:05:00.229878 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.225298    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7dc119-7915-473a-b425-b9fdd217a8e8-workload-socket\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d\" (UID: \"5f7dc119-7915-473a-b425-b9fdd217a8e8\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"
Oct 30 01:05:00.327979 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.327880    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f7dc119-7915-473a-b425-b9fdd217a8e8-istio-token\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d\" (UID: \"5f7dc119-7915-473a-b425-b9fdd217a8e8\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"
Oct 30 01:05:00.327979 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.327961    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f7dc119-7915-473a-b425-b9fdd217a8e8-istiod-ca-cert\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d\" (UID: \"5f7dc119-7915-473a-b425-b9fdd217a8e8\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"
Oct 30 01:05:00.328240 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.328001    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7dc119-7915-473a-b425-b9fdd217a8e8-credential-socket\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d\" (UID: \"5f7dc119-7915-473a-b425-b9fdd217a8e8\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"
Oct 30 01:05:00.328240 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.328030    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f7dc119-7915-473a-b425-b9fdd217a8e8-istio-podinfo\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d\" (UID: \"5f7dc119-7915-473a-b425-b9fdd217a8e8\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"
Oct 30 01:05:00.328240 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.328078    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f7dc119-7915-473a-b425-b9fdd217a8e8-istio-envoy\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d\" (UID: \"5f7dc119-7915-473a-b425-b9fdd217a8e8\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"
Oct 30 01:05:00.328240 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.328148    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7dc119-7915-473a-b425-b9fdd217a8e8-workload-socket\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d\" (UID: \"5f7dc119-7915-473a-b425-b9fdd217a8e8\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"
Oct 30 01:05:00.328456 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.328261    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f7dc119-7915-473a-b425-b9fdd217a8e8-workload-certs\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d\" (UID: \"5f7dc119-7915-473a-b425-b9fdd217a8e8\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"
Oct 30 01:05:00.328456 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.328287    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzfph\" (UniqueName: \"kubernetes.io/projected/5f7dc119-7915-473a-b425-b9fdd217a8e8-kube-api-access-fzfph\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d\" (UID: \"5f7dc119-7915-473a-b425-b9fdd217a8e8\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"
Oct 30 01:05:00.328456 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.328306    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f7dc119-7915-473a-b425-b9fdd217a8e8-istio-data\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d\" (UID: \"5f7dc119-7915-473a-b425-b9fdd217a8e8\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"
Oct 30 01:05:00.328729 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.328699    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f7dc119-7915-473a-b425-b9fdd217a8e8-istiod-ca-cert\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d\" (UID: \"5f7dc119-7915-473a-b425-b9fdd217a8e8\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"
Oct 30 01:05:00.329067 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.329025    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7dc119-7915-473a-b425-b9fdd217a8e8-workload-socket\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d\" (UID: \"5f7dc119-7915-473a-b425-b9fdd217a8e8\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"
Oct 30 01:05:00.329067 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.329062    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f7dc119-7915-473a-b425-b9fdd217a8e8-istio-data\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d\" (UID: \"5f7dc119-7915-473a-b425-b9fdd217a8e8\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"
Oct 30 01:05:00.329226 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.329072    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f7dc119-7915-473a-b425-b9fdd217a8e8-workload-certs\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d\" (UID: \"5f7dc119-7915-473a-b425-b9fdd217a8e8\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"
Oct 30 01:05:00.329226 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.329186    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7dc119-7915-473a-b425-b9fdd217a8e8-credential-socket\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d\" (UID: \"5f7dc119-7915-473a-b425-b9fdd217a8e8\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"
Oct 30 01:05:00.331286 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.331260    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f7dc119-7915-473a-b425-b9fdd217a8e8-istio-podinfo\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d\" (UID: \"5f7dc119-7915-473a-b425-b9fdd217a8e8\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"
Oct 30 01:05:00.331475 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.331458    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f7dc119-7915-473a-b425-b9fdd217a8e8-istio-envoy\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d\" (UID: \"5f7dc119-7915-473a-b425-b9fdd217a8e8\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"
Oct 30 01:05:00.352247 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.352218    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzfph\" (UniqueName: \"kubernetes.io/projected/5f7dc119-7915-473a-b425-b9fdd217a8e8-kube-api-access-fzfph\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d\" (UID: \"5f7dc119-7915-473a-b425-b9fdd217a8e8\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"
Oct 30 01:05:00.352698 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.352677    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f7dc119-7915-473a-b425-b9fdd217a8e8-istio-token\") pod \"e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d\" (UID: \"5f7dc119-7915-473a-b425-b9fdd217a8e8\") " pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"
Oct 30 01:05:00.377220 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.377166    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"
Oct 30 01:05:00.736070 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.736038    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"]
Oct 30 01:05:00.740994 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:05:00.740965    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f7dc119_7915_473a_b425_b9fdd217a8e8.slice/crio-cbc9fdff77178b0b0d4cdd61ea6b614e013233a531e6c2b1d59693bb92ae7a03 WatchSource:0}: Error finding container cbc9fdff77178b0b0d4cdd61ea6b614e013233a531e6c2b1d59693bb92ae7a03: Status 404 returned error can't find the container with id cbc9fdff77178b0b0d4cdd61ea6b614e013233a531e6c2b1d59693bb92ae7a03
Oct 30 01:05:00.744113 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.744075    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:05:00.744214 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.744144    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:05:00.744214 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:00.744179    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:05:01.065390 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:01.065287    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d" event={"ID":"5f7dc119-7915-473a-b425-b9fdd217a8e8","Type":"ContainerStarted","Data":"9f43ac339255b8065d3f28e2906c10d443fe70595a7a00ae80e0b501d6797bf5"}
Oct 30 01:05:01.065390 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:01.065353    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d" event={"ID":"5f7dc119-7915-473a-b425-b9fdd217a8e8","Type":"ContainerStarted","Data":"cbc9fdff77178b0b0d4cdd61ea6b614e013233a531e6c2b1d59693bb92ae7a03"}
Oct 30 01:05:01.134511 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:01.134428    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d" podStartSLOduration=1.134403506 podStartE2EDuration="1.134403506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:01.1099684 +0000 UTC m=+1991.497644789" watchObservedRunningTime="2026-10-30 01:05:01.134403506 +0000 UTC m=+1991.522079898"
Oct 30 01:05:01.137066 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:01.137034    4393 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 01:05:01.137249 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:01.137092    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc" podUID="123708b7-bba7-4cc1-a2c0-9f3887a742f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 01:05:01.378187 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:01.378148    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"
Oct 30 01:05:01.379554 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:01.379525    4393 patch_prober.go:29] interesting pod/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 01:05:01.379698 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:01.379581    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d" podUID="5f7dc119-7915-473a-b425-b9fdd217a8e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 01:05:01.758394 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:01.758300    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s" podUID="01412f93-3e85-41e2-aa25-fb8be317b84b" containerName="main" containerID="cri-o://514475b867f8488b96281093e5d3890dcb65504ac3cdf41d4a4a800ac1d3629f" gracePeriod=45
Oct 30 01:05:02.078907 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:02.078826    4393 generic.go:356] "Generic (PLEG): container finished" podID="01412f93-3e85-41e2-aa25-fb8be317b84b" containerID="514475b867f8488b96281093e5d3890dcb65504ac3cdf41d4a4a800ac1d3629f" exitCode=0
Oct 30 01:05:02.079918 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:02.079895    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s" event={"ID":"01412f93-3e85-41e2-aa25-fb8be317b84b","Type":"ContainerDied","Data":"514475b867f8488b96281093e5d3890dcb65504ac3cdf41d4a4a800ac1d3629f"}
Oct 30 01:05:02.160824 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:02.159952    4393 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 01:05:02.160824 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:02.160018    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc" podUID="123708b7-bba7-4cc1-a2c0-9f3887a742f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 01:05:02.258153 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:02.258123    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s"
Oct 30 01:05:02.373740 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:02.373714    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01412f93-3e85-41e2-aa25-fb8be317b84b-home\" (UniqueName: \"kubernetes.io/empty-dir/01412f93-3e85-41e2-aa25-fb8be317b84b-home\") pod \"01412f93-3e85-41e2-aa25-fb8be317b84b\" (UID: \"01412f93-3e85-41e2-aa25-fb8be317b84b\") "
Oct 30 01:05:02.373931 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:02.373876    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01412f93-3e85-41e2-aa25-fb8be317b84b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01412f93-3e85-41e2-aa25-fb8be317b84b-model-cache\") pod \"01412f93-3e85-41e2-aa25-fb8be317b84b\" (UID: \"01412f93-3e85-41e2-aa25-fb8be317b84b\") "
Oct 30 01:05:02.373931 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:02.373917    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01412f93-3e85-41e2-aa25-fb8be317b84b-tls-certs\" (UniqueName: \"kubernetes.io/secret/01412f93-3e85-41e2-aa25-fb8be317b84b-tls-certs\") pod \"01412f93-3e85-41e2-aa25-fb8be317b84b\" (UID: \"01412f93-3e85-41e2-aa25-fb8be317b84b\") "
Oct 30 01:05:02.374160 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:02.374129    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01412f93-3e85-41e2-aa25-fb8be317b84b-kube-api-access-stjjl\" (UniqueName: \"kubernetes.io/projected/01412f93-3e85-41e2-aa25-fb8be317b84b-kube-api-access-stjjl\") pod \"01412f93-3e85-41e2-aa25-fb8be317b84b\" (UID: \"01412f93-3e85-41e2-aa25-fb8be317b84b\") "
Oct 30 01:05:02.374242 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:02.374224    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01412f93-3e85-41e2-aa25-fb8be317b84b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/01412f93-3e85-41e2-aa25-fb8be317b84b-dshm\") pod \"01412f93-3e85-41e2-aa25-fb8be317b84b\" (UID: \"01412f93-3e85-41e2-aa25-fb8be317b84b\") "
Oct 30 01:05:02.374242 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:02.374219    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01412f93-3e85-41e2-aa25-fb8be317b84b-home" pod "01412f93-3e85-41e2-aa25-fb8be317b84b" (UID: "01412f93-3e85-41e2-aa25-fb8be317b84b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:02.374347 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:02.374241    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01412f93-3e85-41e2-aa25-fb8be317b84b-model-cache" pod "01412f93-3e85-41e2-aa25-fb8be317b84b" (UID: "01412f93-3e85-41e2-aa25-fb8be317b84b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:02.374400 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:02.374351    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01412f93-3e85-41e2-aa25-fb8be317b84b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01412f93-3e85-41e2-aa25-fb8be317b84b-tmp-dir\") pod \"01412f93-3e85-41e2-aa25-fb8be317b84b\" (UID: \"01412f93-3e85-41e2-aa25-fb8be317b84b\") "
Oct 30 01:05:02.375042 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:02.375016    4393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01412f93-3e85-41e2-aa25-fb8be317b84b-home\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:05:02.375042 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:02.375041    4393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01412f93-3e85-41e2-aa25-fb8be317b84b-model-cache\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:05:02.375301 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:02.375281    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01412f93-3e85-41e2-aa25-fb8be317b84b-tmp-dir" pod "01412f93-3e85-41e2-aa25-fb8be317b84b" (UID: "01412f93-3e85-41e2-aa25-fb8be317b84b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:02.378239 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:02.377627    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01412f93-3e85-41e2-aa25-fb8be317b84b-kube-api-access-stjjl" pod "01412f93-3e85-41e2-aa25-fb8be317b84b" (UID: "01412f93-3e85-41e2-aa25-fb8be317b84b"). InnerVolumeSpecName "kube-api-access-stjjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:02.378239 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:02.377635    4393 patch_prober.go:29] interesting pod/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 01:05:02.378239 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:02.377722    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d" podUID="5f7dc119-7915-473a-b425-b9fdd217a8e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 01:05:02.379734 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:02.379673    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01412f93-3e85-41e2-aa25-fb8be317b84b-dshm" pod "01412f93-3e85-41e2-aa25-fb8be317b84b" (UID: "01412f93-3e85-41e2-aa25-fb8be317b84b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:02.380933 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:02.380895    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01412f93-3e85-41e2-aa25-fb8be317b84b-tls-certs" pod "01412f93-3e85-41e2-aa25-fb8be317b84b" (UID: "01412f93-3e85-41e2-aa25-fb8be317b84b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:05:02.476851 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:02.476815    4393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01412f93-3e85-41e2-aa25-fb8be317b84b-tls-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:05:02.476851 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:02.476848    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stjjl\" (UniqueName: \"kubernetes.io/projected/01412f93-3e85-41e2-aa25-fb8be317b84b-kube-api-access-stjjl\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:05:02.476851 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:02.476858    4393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01412f93-3e85-41e2-aa25-fb8be317b84b-dshm\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:05:02.477128 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:02.476866    4393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01412f93-3e85-41e2-aa25-fb8be317b84b-tmp-dir\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:05:03.093004 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:03.092972    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s" event={"ID":"01412f93-3e85-41e2-aa25-fb8be317b84b","Type":"ContainerDied","Data":"2a1da990a78fb3aecff35ecea05c47c019317ea00810dd0eaae317900238785a"}
Oct 30 01:05:03.093473 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:03.093021    4393 scope.go:122] "RemoveContainer" containerID="514475b867f8488b96281093e5d3890dcb65504ac3cdf41d4a4a800ac1d3629f"
Oct 30 01:05:03.093687 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:03.093665    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s"
Oct 30 01:05:03.122806 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:03.119022    4393 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 01:05:03.122806 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:03.119084    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc" podUID="123708b7-bba7-4cc1-a2c0-9f3887a742f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 01:05:03.200356 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:03.200322    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s"]
Oct 30 01:05:03.263866 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:03.263829    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-cce513/auth-test-model-cce513-kserve-796cd476f7-x9b8s"]
Oct 30 01:05:03.386761 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:03.380192    4393 patch_prober.go:29] interesting pod/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 01:05:03.386761 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:03.380266    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d" podUID="5f7dc119-7915-473a-b425-b9fdd217a8e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 01:05:03.765124 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:03.765043    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01412f93-3e85-41e2-aa25-fb8be317b84b" path="/var/lib/kubelet/pods/01412f93-3e85-41e2-aa25-fb8be317b84b/volumes"
Oct 30 01:05:04.121864 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:04.110133    4393 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 01:05:04.121864 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:04.110195    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc" podUID="123708b7-bba7-4cc1-a2c0-9f3887a742f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 01:05:04.378296 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:04.378222    4393 patch_prober.go:29] interesting pod/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 01:05:04.378296 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:04.378274    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d" podUID="5f7dc119-7915-473a-b425-b9fdd217a8e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 01:05:05.121709 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:05.121675    4393 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 01:05:05.126146 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:05.126114    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc" podUID="123708b7-bba7-4cc1-a2c0-9f3887a742f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 01:05:05.378820 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:05.378435    4393 patch_prober.go:29] interesting pod/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 01:05:05.378820 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:05.378499    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d" podUID="5f7dc119-7915-473a-b425-b9fdd217a8e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 01:05:06.112805 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:06.112011    4393 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 30 01:05:06.112805 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:06.112071    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc" podUID="123708b7-bba7-4cc1-a2c0-9f3887a742f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 30 01:05:06.378359 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:06.378322    4393 patch_prober.go:29] interesting pod/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 01:05:06.378863 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:06.378384    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d" podUID="5f7dc119-7915-473a-b425-b9fdd217a8e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 01:05:06.667803 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:06.667694    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb" podUID="0b136c12-ff27-4e1a-8f06-786770f0cb4a" containerName="main" containerID="cri-o://025c293fdb1e6aa5ef164785dce656a16fea3045b0974070d55c4253f4d862a9" gracePeriod=45
Oct 30 01:05:06.992131 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:06.992104    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb"
Oct 30 01:05:07.036524 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:07.036486    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b136c12-ff27-4e1a-8f06-786770f0cb4a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b136c12-ff27-4e1a-8f06-786770f0cb4a-model-cache\") pod \"0b136c12-ff27-4e1a-8f06-786770f0cb4a\" (UID: \"0b136c12-ff27-4e1a-8f06-786770f0cb4a\") "
Oct 30 01:05:07.036706 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:07.036612    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b136c12-ff27-4e1a-8f06-786770f0cb4a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b136c12-ff27-4e1a-8f06-786770f0cb4a-dshm\") pod \"0b136c12-ff27-4e1a-8f06-786770f0cb4a\" (UID: \"0b136c12-ff27-4e1a-8f06-786770f0cb4a\") "
Oct 30 01:05:07.036706 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:07.036636    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b136c12-ff27-4e1a-8f06-786770f0cb4a-kube-api-access-vbb9z\" (UniqueName: \"kubernetes.io/projected/0b136c12-ff27-4e1a-8f06-786770f0cb4a-kube-api-access-vbb9z\") pod \"0b136c12-ff27-4e1a-8f06-786770f0cb4a\" (UID: \"0b136c12-ff27-4e1a-8f06-786770f0cb4a\") "
Oct 30 01:05:07.036706 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:07.036686    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b136c12-ff27-4e1a-8f06-786770f0cb4a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b136c12-ff27-4e1a-8f06-786770f0cb4a-tmp-dir\") pod \"0b136c12-ff27-4e1a-8f06-786770f0cb4a\" (UID: \"0b136c12-ff27-4e1a-8f06-786770f0cb4a\") "
Oct 30 01:05:07.036944 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:07.036725    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b136c12-ff27-4e1a-8f06-786770f0cb4a-tls-certs\" (UniqueName: \"kubernetes.io/secret/0b136c12-ff27-4e1a-8f06-786770f0cb4a-tls-certs\") pod \"0b136c12-ff27-4e1a-8f06-786770f0cb4a\" (UID: \"0b136c12-ff27-4e1a-8f06-786770f0cb4a\") "
Oct 30 01:05:07.036944 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:07.036797    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b136c12-ff27-4e1a-8f06-786770f0cb4a-home\" (UniqueName: \"kubernetes.io/empty-dir/0b136c12-ff27-4e1a-8f06-786770f0cb4a-home\") pod \"0b136c12-ff27-4e1a-8f06-786770f0cb4a\" (UID: \"0b136c12-ff27-4e1a-8f06-786770f0cb4a\") "
Oct 30 01:05:07.036944 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:07.036817    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b136c12-ff27-4e1a-8f06-786770f0cb4a-model-cache" pod "0b136c12-ff27-4e1a-8f06-786770f0cb4a" (UID: "0b136c12-ff27-4e1a-8f06-786770f0cb4a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:07.037110 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:07.037031    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b136c12-ff27-4e1a-8f06-786770f0cb4a-tmp-dir" pod "0b136c12-ff27-4e1a-8f06-786770f0cb4a" (UID: "0b136c12-ff27-4e1a-8f06-786770f0cb4a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:07.037329 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:07.037292    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b136c12-ff27-4e1a-8f06-786770f0cb4a-home" pod "0b136c12-ff27-4e1a-8f06-786770f0cb4a" (UID: "0b136c12-ff27-4e1a-8f06-786770f0cb4a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:07.037619 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:07.037486    4393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b136c12-ff27-4e1a-8f06-786770f0cb4a-model-cache\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:05:07.037619 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:07.037504    4393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b136c12-ff27-4e1a-8f06-786770f0cb4a-tmp-dir\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:05:07.037619 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:07.037512    4393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b136c12-ff27-4e1a-8f06-786770f0cb4a-home\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:05:07.039387 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:07.039360    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b136c12-ff27-4e1a-8f06-786770f0cb4a-kube-api-access-vbb9z" pod "0b136c12-ff27-4e1a-8f06-786770f0cb4a" (UID: "0b136c12-ff27-4e1a-8f06-786770f0cb4a"). InnerVolumeSpecName "kube-api-access-vbb9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:07.039677 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:07.039659    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b136c12-ff27-4e1a-8f06-786770f0cb4a-dshm" pod "0b136c12-ff27-4e1a-8f06-786770f0cb4a" (UID: "0b136c12-ff27-4e1a-8f06-786770f0cb4a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:07.039726 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:07.039666    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b136c12-ff27-4e1a-8f06-786770f0cb4a-tls-certs" pod "0b136c12-ff27-4e1a-8f06-786770f0cb4a" (UID: "0b136c12-ff27-4e1a-8f06-786770f0cb4a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:05:07.142827 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:07.135802    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"
Oct 30 01:05:07.142827 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:07.137308    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"
Oct 30 01:05:07.142827 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:07.139042    4393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b136c12-ff27-4e1a-8f06-786770f0cb4a-dshm\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:05:07.142827 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:07.139071    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbb9z\" (UniqueName: \"kubernetes.io/projected/0b136c12-ff27-4e1a-8f06-786770f0cb4a-kube-api-access-vbb9z\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:05:07.142827 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:07.139090    4393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b136c12-ff27-4e1a-8f06-786770f0cb4a-tls-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:05:07.142827 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:07.139168    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"
Oct 30 01:05:07.182503 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:07.182474    4393 generic.go:356] "Generic (PLEG): container finished" podID="0b136c12-ff27-4e1a-8f06-786770f0cb4a" containerID="025c293fdb1e6aa5ef164785dce656a16fea3045b0974070d55c4253f4d862a9" exitCode=0
Oct 30 01:05:07.182693 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:07.182524    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb"
Oct 30 01:05:07.182693 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:07.182569    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb" event={"ID":"0b136c12-ff27-4e1a-8f06-786770f0cb4a","Type":"ContainerDied","Data":"025c293fdb1e6aa5ef164785dce656a16fea3045b0974070d55c4253f4d862a9"}
Oct 30 01:05:07.182693 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:07.182618    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb" event={"ID":"0b136c12-ff27-4e1a-8f06-786770f0cb4a","Type":"ContainerDied","Data":"8c23d2cbaf1ca1cbaabb8cca39fbbf974b9292e359f9031b686c4c531d61f79a"}
Oct 30 01:05:07.182693 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:07.182637    4393 scope.go:122] "RemoveContainer" containerID="025c293fdb1e6aa5ef164785dce656a16fea3045b0974070d55c4253f4d862a9"
Oct 30 01:05:07.195523 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:07.195498    4393 scope.go:122] "RemoveContainer" containerID="025c293fdb1e6aa5ef164785dce656a16fea3045b0974070d55c4253f4d862a9"
Oct 30 01:05:07.195830 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:05:07.195805    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"025c293fdb1e6aa5ef164785dce656a16fea3045b0974070d55c4253f4d862a9\": container with ID starting with 025c293fdb1e6aa5ef164785dce656a16fea3045b0974070d55c4253f4d862a9 not found: ID does not exist" containerID="025c293fdb1e6aa5ef164785dce656a16fea3045b0974070d55c4253f4d862a9"
Oct 30 01:05:07.195918 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:07.195838    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"025c293fdb1e6aa5ef164785dce656a16fea3045b0974070d55c4253f4d862a9"} err="failed to get container status \"025c293fdb1e6aa5ef164785dce656a16fea3045b0974070d55c4253f4d862a9\": rpc error: code = NotFound desc = could not find container \"025c293fdb1e6aa5ef164785dce656a16fea3045b0974070d55c4253f4d862a9\": container with ID starting with 025c293fdb1e6aa5ef164785dce656a16fea3045b0974070d55c4253f4d862a9 not found: ID does not exist"
Oct 30 01:05:07.266980 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:07.266883    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb"]
Oct 30 01:05:07.275460 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:07.275429    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-5007e1/test-model-5007e1-kserve-6d4b6f4955-hw4jb"]
Oct 30 01:05:07.378356 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:07.378313    4393 patch_prober.go:29] interesting pod/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 01:05:07.378891 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:07.378367    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d" podUID="5f7dc119-7915-473a-b425-b9fdd217a8e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 01:05:07.728187 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:07.728140    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b136c12-ff27-4e1a-8f06-786770f0cb4a" path="/var/lib/kubelet/pods/0b136c12-ff27-4e1a-8f06-786770f0cb4a/volumes"
Oct 30 01:05:08.382577 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:08.382538    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"
Oct 30 01:05:08.383053 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:08.383040    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"
Oct 30 01:05:08.383753 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:08.383730    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"
Oct 30 01:05:23.075033 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:23.074995    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p"]
Oct 30 01:05:23.075542 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:23.075224    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p" podUID="ace5c083-c41b-46bd-9a0e-5545c4e141ea" containerName="maas-api" containerID="cri-o://fa9f3c0926f83a402ddd4bcb866337eac01a1395bc687e1e6009ac7d3be98498" gracePeriod=30
Oct 30 01:05:23.452933 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:23.452684    4393 generic.go:356] "Generic (PLEG): container finished" podID="ace5c083-c41b-46bd-9a0e-5545c4e141ea" containerID="fa9f3c0926f83a402ddd4bcb866337eac01a1395bc687e1e6009ac7d3be98498" exitCode=0
Oct 30 01:05:23.453117 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:23.452932    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p" event={"ID":"ace5c083-c41b-46bd-9a0e-5545c4e141ea","Type":"ContainerDied","Data":"fa9f3c0926f83a402ddd4bcb866337eac01a1395bc687e1e6009ac7d3be98498"}
Oct 30 01:05:23.564802 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:23.560825    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p"
Oct 30 01:05:23.664808 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:23.664484    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ace5c083-c41b-46bd-9a0e-5545c4e141ea-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ace5c083-c41b-46bd-9a0e-5545c4e141ea-maas-api-tls\") pod \"ace5c083-c41b-46bd-9a0e-5545c4e141ea\" (UID: \"ace5c083-c41b-46bd-9a0e-5545c4e141ea\") "
Oct 30 01:05:23.664808 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:23.664797    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ace5c083-c41b-46bd-9a0e-5545c4e141ea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ace5c083-c41b-46bd-9a0e-5545c4e141ea-metrics-certs\") pod \"ace5c083-c41b-46bd-9a0e-5545c4e141ea\" (UID: \"ace5c083-c41b-46bd-9a0e-5545c4e141ea\") "
Oct 30 01:05:23.665067 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:23.664860    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ace5c083-c41b-46bd-9a0e-5545c4e141ea-kube-api-access-8p74h\" (UniqueName: \"kubernetes.io/projected/ace5c083-c41b-46bd-9a0e-5545c4e141ea-kube-api-access-8p74h\") pod \"ace5c083-c41b-46bd-9a0e-5545c4e141ea\" (UID: \"ace5c083-c41b-46bd-9a0e-5545c4e141ea\") "
Oct 30 01:05:23.665067 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:23.664913    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ace5c083-c41b-46bd-9a0e-5545c4e141ea-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ace5c083-c41b-46bd-9a0e-5545c4e141ea-openshift-service-ca\") pod \"ace5c083-c41b-46bd-9a0e-5545c4e141ea\" (UID: \"ace5c083-c41b-46bd-9a0e-5545c4e141ea\") "
Oct 30 01:05:23.673804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:23.666353    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ace5c083-c41b-46bd-9a0e-5545c4e141ea-openshift-service-ca" pod "ace5c083-c41b-46bd-9a0e-5545c4e141ea" (UID: "ace5c083-c41b-46bd-9a0e-5545c4e141ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:05:23.755536 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:23.755497    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ace5c083-c41b-46bd-9a0e-5545c4e141ea-maas-api-tls" pod "ace5c083-c41b-46bd-9a0e-5545c4e141ea" (UID: "ace5c083-c41b-46bd-9a0e-5545c4e141ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:05:23.756559 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:23.756529    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ace5c083-c41b-46bd-9a0e-5545c4e141ea-kube-api-access-8p74h" pod "ace5c083-c41b-46bd-9a0e-5545c4e141ea" (UID: "ace5c083-c41b-46bd-9a0e-5545c4e141ea"). InnerVolumeSpecName "kube-api-access-8p74h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:23.758581 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:23.758522    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ace5c083-c41b-46bd-9a0e-5545c4e141ea-metrics-certs" pod "ace5c083-c41b-46bd-9a0e-5545c4e141ea" (UID: "ace5c083-c41b-46bd-9a0e-5545c4e141ea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:05:23.774108 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:23.773062    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ace5c083-c41b-46bd-9a0e-5545c4e141ea-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:05:23.774108 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:23.773094    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ace5c083-c41b-46bd-9a0e-5545c4e141ea-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:05:23.774108 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:23.773110    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p74h\" (UniqueName: \"kubernetes.io/projected/ace5c083-c41b-46bd-9a0e-5545c4e141ea-kube-api-access-8p74h\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:05:23.774108 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:23.773125    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ace5c083-c41b-46bd-9a0e-5545c4e141ea-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:05:24.467670 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:24.467574    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p" event={"ID":"ace5c083-c41b-46bd-9a0e-5545c4e141ea","Type":"ContainerDied","Data":"aa543d719f1cf478e820ebe69d2240e1a9e7543d7574602a5e4723d7a3af6a98"}
Oct 30 01:05:24.468207 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:24.467715    4393 scope.go:122] "RemoveContainer" containerID="fa9f3c0926f83a402ddd4bcb866337eac01a1395bc687e1e6009ac7d3be98498"
Oct 30 01:05:24.468207 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:24.467968    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p"
Oct 30 01:05:24.512974 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:24.512941    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p"]
Oct 30 01:05:24.524831 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:24.524762    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-84b89f-7f8cbb87cf-h6b4p"]
Oct 30 01:05:25.726963 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:25.726918    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ace5c083-c41b-46bd-9a0e-5545c4e141ea" path="/var/lib/kubelet/pods/ace5c083-c41b-46bd-9a0e-5545c4e141ea/volumes"
Oct 30 01:05:26.512009 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:05:26.511973    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:05:29.477844 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:29.477807    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md"]
Oct 30 01:05:29.478639 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:29.478623    4393 cpu_manager.go:414] "containerMap: removing container" podUID="01412f93-3e85-41e2-aa25-fb8be317b84b" containerName="main"
Oct 30 01:05:29.478639 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:29.478639    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01412f93-3e85-41e2-aa25-fb8be317b84b" containerName="main"
Oct 30 01:05:29.478721 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:29.478659    4393 cpu_manager.go:414] "containerMap: removing container" podUID="0b136c12-ff27-4e1a-8f06-786770f0cb4a" containerName="main"
Oct 30 01:05:29.478721 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:29.478665    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b136c12-ff27-4e1a-8f06-786770f0cb4a" containerName="main"
Oct 30 01:05:29.478721 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:29.478680    4393 cpu_manager.go:414] "containerMap: removing container" podUID="ace5c083-c41b-46bd-9a0e-5545c4e141ea" containerName="maas-api"
Oct 30 01:05:29.478721 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:29.478685    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ace5c083-c41b-46bd-9a0e-5545c4e141ea" containerName="maas-api"
Oct 30 01:05:29.478917 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:29.478828    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="01412f93-3e85-41e2-aa25-fb8be317b84b" containerName="main"
Oct 30 01:05:29.478917 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:29.478838    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b136c12-ff27-4e1a-8f06-786770f0cb4a" containerName="main"
Oct 30 01:05:29.478917 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:29.478847    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="ace5c083-c41b-46bd-9a0e-5545c4e141ea" containerName="maas-api"
Oct 30 01:05:29.484882 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:29.484859    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md"
Oct 30 01:05:29.517242 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:29.517216    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md"]
Oct 30 01:05:29.556499 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:29.556460    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-009e78-serving-cert\""
Oct 30 01:05:29.572687 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:29.572651    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7028bbc0-dd7e-4525-a1db-8bf264d8ad58-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md\" (UID: \"7028bbc0-dd7e-4525-a1db-8bf264d8ad58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md"
Oct 30 01:05:29.572881 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:29.572792    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7028bbc0-dd7e-4525-a1db-8bf264d8ad58-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md\" (UID: \"7028bbc0-dd7e-4525-a1db-8bf264d8ad58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md"
Oct 30 01:05:29.573034 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:29.573010    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chx5r\" (UniqueName: \"kubernetes.io/projected/7028bbc0-dd7e-4525-a1db-8bf264d8ad58-kube-api-access-chx5r\") pod \"maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md\" (UID: \"7028bbc0-dd7e-4525-a1db-8bf264d8ad58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md"
Oct 30 01:05:29.573169 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:29.573146    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7028bbc0-dd7e-4525-a1db-8bf264d8ad58-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md\" (UID: \"7028bbc0-dd7e-4525-a1db-8bf264d8ad58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md"
Oct 30 01:05:29.674428 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:29.674393    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chx5r\" (UniqueName: \"kubernetes.io/projected/7028bbc0-dd7e-4525-a1db-8bf264d8ad58-kube-api-access-chx5r\") pod \"maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md\" (UID: \"7028bbc0-dd7e-4525-a1db-8bf264d8ad58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md"
Oct 30 01:05:29.674610 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:29.674484    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7028bbc0-dd7e-4525-a1db-8bf264d8ad58-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md\" (UID: \"7028bbc0-dd7e-4525-a1db-8bf264d8ad58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md"
Oct 30 01:05:29.674677 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:29.674636    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7028bbc0-dd7e-4525-a1db-8bf264d8ad58-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md\" (UID: \"7028bbc0-dd7e-4525-a1db-8bf264d8ad58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md"
Oct 30 01:05:29.674734 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:29.674706    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7028bbc0-dd7e-4525-a1db-8bf264d8ad58-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md\" (UID: \"7028bbc0-dd7e-4525-a1db-8bf264d8ad58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md"
Oct 30 01:05:29.675199 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:29.675163    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7028bbc0-dd7e-4525-a1db-8bf264d8ad58-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md\" (UID: \"7028bbc0-dd7e-4525-a1db-8bf264d8ad58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md"
Oct 30 01:05:29.677315 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:29.677285    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7028bbc0-dd7e-4525-a1db-8bf264d8ad58-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md\" (UID: \"7028bbc0-dd7e-4525-a1db-8bf264d8ad58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md"
Oct 30 01:05:29.677447 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:29.677327    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7028bbc0-dd7e-4525-a1db-8bf264d8ad58-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md\" (UID: \"7028bbc0-dd7e-4525-a1db-8bf264d8ad58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md"
Oct 30 01:05:29.697491 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:29.697464    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chx5r\" (UniqueName: \"kubernetes.io/projected/7028bbc0-dd7e-4525-a1db-8bf264d8ad58-kube-api-access-chx5r\") pod \"maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md\" (UID: \"7028bbc0-dd7e-4525-a1db-8bf264d8ad58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md"
Oct 30 01:05:29.839284 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:29.839190    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md"
Oct 30 01:05:30.156310 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:05:30.156275    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7028bbc0_dd7e_4525_a1db_8bf264d8ad58.slice/crio-64eecb4bd7c32dbe76a85554fdb4ba39592f87c92c541315a64fcbf27d422511 WatchSource:0}: Error finding container 64eecb4bd7c32dbe76a85554fdb4ba39592f87c92c541315a64fcbf27d422511: Status 404 returned error can't find the container with id 64eecb4bd7c32dbe76a85554fdb4ba39592f87c92c541315a64fcbf27d422511
Oct 30 01:05:30.156503 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:30.156484    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md"]
Oct 30 01:05:30.157718 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:30.157699    4393 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 01:05:30.521475 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:30.521383    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md" event={"ID":"7028bbc0-dd7e-4525-a1db-8bf264d8ad58","Type":"ContainerStarted","Data":"64eecb4bd7c32dbe76a85554fdb4ba39592f87c92c541315a64fcbf27d422511"}
Oct 30 01:05:31.666513 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:31.666144    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md" event={"ID":"7028bbc0-dd7e-4525-a1db-8bf264d8ad58","Type":"ContainerStarted","Data":"ea7a05c5bd5662b0a96f950b902237681c33f95c0066beb4a4eec5e717690dd0"}
Oct 30 01:05:31.667471 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:31.667447    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md"
Oct 30 01:05:31.732489 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:31.731250    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md" podStartSLOduration=2.366445276 podStartE2EDuration="2.731230876s" totalImagesPullingTime="364.7856ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:31.729184188 +0000 UTC m=+2022.116860578" watchObservedRunningTime="2026-10-30 01:05:31.731230876 +0000 UTC m=+2022.118907267"
Oct 30 01:05:32.860215 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:32.860178    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg"]
Oct 30 01:05:32.860923 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:32.860415    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg" podUID="98869821-d2ad-490b-b71b-ea9d17af6327" containerName="maas-api" containerID="cri-o://e9bb9da24f9ddea11c331518ac564de0900a6588d3405cd7d8ca5aeb324322be" gracePeriod=30
Oct 30 01:05:33.331803 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:33.328188    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg"
Oct 30 01:05:33.398801 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:33.390590    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98869821-d2ad-490b-b71b-ea9d17af6327-kube-api-access-r87tk\" (UniqueName: \"kubernetes.io/projected/98869821-d2ad-490b-b71b-ea9d17af6327-kube-api-access-r87tk\") pod \"98869821-d2ad-490b-b71b-ea9d17af6327\" (UID: \"98869821-d2ad-490b-b71b-ea9d17af6327\") "
Oct 30 01:05:33.398801 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:33.390770    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98869821-d2ad-490b-b71b-ea9d17af6327-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98869821-d2ad-490b-b71b-ea9d17af6327-openshift-service-ca\") pod \"98869821-d2ad-490b-b71b-ea9d17af6327\" (UID: \"98869821-d2ad-490b-b71b-ea9d17af6327\") "
Oct 30 01:05:33.398801 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:33.390831    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98869821-d2ad-490b-b71b-ea9d17af6327-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98869821-d2ad-490b-b71b-ea9d17af6327-maas-api-tls\") pod \"98869821-d2ad-490b-b71b-ea9d17af6327\" (UID: \"98869821-d2ad-490b-b71b-ea9d17af6327\") "
Oct 30 01:05:33.398801 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:33.390911    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98869821-d2ad-490b-b71b-ea9d17af6327-metrics-certs\" (UniqueName: \"kubernetes.io/secret/98869821-d2ad-490b-b71b-ea9d17af6327-metrics-certs\") pod \"98869821-d2ad-490b-b71b-ea9d17af6327\" (UID: \"98869821-d2ad-490b-b71b-ea9d17af6327\") "
Oct 30 01:05:33.398801 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:33.392352    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98869821-d2ad-490b-b71b-ea9d17af6327-openshift-service-ca" pod "98869821-d2ad-490b-b71b-ea9d17af6327" (UID: "98869821-d2ad-490b-b71b-ea9d17af6327"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:05:33.405212 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:33.405165    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98869821-d2ad-490b-b71b-ea9d17af6327-metrics-certs" pod "98869821-d2ad-490b-b71b-ea9d17af6327" (UID: "98869821-d2ad-490b-b71b-ea9d17af6327"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:05:33.512351 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:33.512249    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98869821-d2ad-490b-b71b-ea9d17af6327-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:05:33.512351 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:33.512296    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98869821-d2ad-490b-b71b-ea9d17af6327-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:05:33.521271 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:33.521208    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98869821-d2ad-490b-b71b-ea9d17af6327-kube-api-access-r87tk" pod "98869821-d2ad-490b-b71b-ea9d17af6327" (UID: "98869821-d2ad-490b-b71b-ea9d17af6327"). InnerVolumeSpecName "kube-api-access-r87tk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:33.554612 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:33.554480    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98869821-d2ad-490b-b71b-ea9d17af6327-maas-api-tls" pod "98869821-d2ad-490b-b71b-ea9d17af6327" (UID: "98869821-d2ad-490b-b71b-ea9d17af6327"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:05:33.614380 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:33.614347    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98869821-d2ad-490b-b71b-ea9d17af6327-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:05:33.614380 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:33.614381    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r87tk\" (UniqueName: \"kubernetes.io/projected/98869821-d2ad-490b-b71b-ea9d17af6327-kube-api-access-r87tk\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:05:33.684341 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:33.684304    4393 generic.go:356] "Generic (PLEG): container finished" podID="98869821-d2ad-490b-b71b-ea9d17af6327" containerID="e9bb9da24f9ddea11c331518ac564de0900a6588d3405cd7d8ca5aeb324322be" exitCode=0
Oct 30 01:05:33.684522 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:33.684376    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg"
Oct 30 01:05:33.684522 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:33.684396    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg" event={"ID":"98869821-d2ad-490b-b71b-ea9d17af6327","Type":"ContainerDied","Data":"e9bb9da24f9ddea11c331518ac564de0900a6588d3405cd7d8ca5aeb324322be"}
Oct 30 01:05:33.684522 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:33.684481    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg" event={"ID":"98869821-d2ad-490b-b71b-ea9d17af6327","Type":"ContainerDied","Data":"3287a585e5db03c2bf597dde9d3b666ceb218f3c3475749766517100e4a58950"}
Oct 30 01:05:33.684522 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:33.684507    4393 scope.go:122] "RemoveContainer" containerID="e9bb9da24f9ddea11c331518ac564de0900a6588d3405cd7d8ca5aeb324322be"
Oct 30 01:05:33.697201 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:33.697181    4393 scope.go:122] "RemoveContainer" containerID="e9bb9da24f9ddea11c331518ac564de0900a6588d3405cd7d8ca5aeb324322be"
Oct 30 01:05:33.697516 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:05:33.697490    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9bb9da24f9ddea11c331518ac564de0900a6588d3405cd7d8ca5aeb324322be\": container with ID starting with e9bb9da24f9ddea11c331518ac564de0900a6588d3405cd7d8ca5aeb324322be not found: ID does not exist" containerID="e9bb9da24f9ddea11c331518ac564de0900a6588d3405cd7d8ca5aeb324322be"
Oct 30 01:05:33.697608 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:33.697518    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9bb9da24f9ddea11c331518ac564de0900a6588d3405cd7d8ca5aeb324322be"} err="failed to get container status \"e9bb9da24f9ddea11c331518ac564de0900a6588d3405cd7d8ca5aeb324322be\": rpc error: code = NotFound desc = could not find container \"e9bb9da24f9ddea11c331518ac564de0900a6588d3405cd7d8ca5aeb324322be\": container with ID starting with e9bb9da24f9ddea11c331518ac564de0900a6588d3405cd7d8ca5aeb324322be not found: ID does not exist"
Oct 30 01:05:33.742348 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:33.742315    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg"]
Oct 30 01:05:33.746612 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:33.746591    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8d2a70-574c4d44b7-9wsgg"]
Oct 30 01:05:35.741925 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:35.741891    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98869821-d2ad-490b-b71b-ea9d17af6327" path="/var/lib/kubelet/pods/98869821-d2ad-490b-b71b-ea9d17af6327/volumes"
Oct 30 01:05:38.691315 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:38.691281    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md"
Oct 30 01:05:41.108454 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:41.108372    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"]
Oct 30 01:05:41.109619 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:41.109279    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw" podUID="762e23fe-a544-49c1-a734-1c93e8bac082" containerName="istio-proxy" containerID="cri-o://14266b28595a9771a400bc3f01c7907a46f2aed1779a72615e1620a7638d9706" gracePeriod=30
Oct 30 01:05:46.821272 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:46.821244    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"
Oct 30 01:05:46.879291 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:46.879249    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762e23fe-a544-49c1-a734-1c93e8bac082-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762e23fe-a544-49c1-a734-1c93e8bac082-istio-data\") pod \"762e23fe-a544-49c1-a734-1c93e8bac082\" (UID: \"762e23fe-a544-49c1-a734-1c93e8bac082\") "
Oct 30 01:05:46.880429 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:46.879340    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/762e23fe-a544-49c1-a734-1c93e8bac082-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762e23fe-a544-49c1-a734-1c93e8bac082-istiod-ca-cert\") pod \"762e23fe-a544-49c1-a734-1c93e8bac082\" (UID: \"762e23fe-a544-49c1-a734-1c93e8bac082\") "
Oct 30 01:05:46.880429 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:46.879367    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/762e23fe-a544-49c1-a734-1c93e8bac082-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762e23fe-a544-49c1-a734-1c93e8bac082-istio-podinfo\") pod \"762e23fe-a544-49c1-a734-1c93e8bac082\" (UID: \"762e23fe-a544-49c1-a734-1c93e8bac082\") "
Oct 30 01:05:46.880429 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:46.879417    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/762e23fe-a544-49c1-a734-1c93e8bac082-kube-api-access-sg779\" (UniqueName: \"kubernetes.io/projected/762e23fe-a544-49c1-a734-1c93e8bac082-kube-api-access-sg779\") pod \"762e23fe-a544-49c1-a734-1c93e8bac082\" (UID: \"762e23fe-a544-49c1-a734-1c93e8bac082\") "
Oct 30 01:05:46.880429 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:46.879559    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762e23fe-a544-49c1-a734-1c93e8bac082-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762e23fe-a544-49c1-a734-1c93e8bac082-istio-envoy\") pod \"762e23fe-a544-49c1-a734-1c93e8bac082\" (UID: \"762e23fe-a544-49c1-a734-1c93e8bac082\") "
Oct 30 01:05:46.880429 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:46.879578    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762e23fe-a544-49c1-a734-1c93e8bac082-istio-data" pod "762e23fe-a544-49c1-a734-1c93e8bac082" (UID: "762e23fe-a544-49c1-a734-1c93e8bac082"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:46.880429 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:46.879631    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762e23fe-a544-49c1-a734-1c93e8bac082-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762e23fe-a544-49c1-a734-1c93e8bac082-workload-certs\") pod \"762e23fe-a544-49c1-a734-1c93e8bac082\" (UID: \"762e23fe-a544-49c1-a734-1c93e8bac082\") "
Oct 30 01:05:46.880429 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:46.879709    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762e23fe-a544-49c1-a734-1c93e8bac082-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762e23fe-a544-49c1-a734-1c93e8bac082-credential-socket\") pod \"762e23fe-a544-49c1-a734-1c93e8bac082\" (UID: \"762e23fe-a544-49c1-a734-1c93e8bac082\") "
Oct 30 01:05:46.880429 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:46.879759    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/762e23fe-a544-49c1-a734-1c93e8bac082-istio-token\" (UniqueName: \"kubernetes.io/projected/762e23fe-a544-49c1-a734-1c93e8bac082-istio-token\") pod \"762e23fe-a544-49c1-a734-1c93e8bac082\" (UID: \"762e23fe-a544-49c1-a734-1c93e8bac082\") "
Oct 30 01:05:46.880429 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:46.879804    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/762e23fe-a544-49c1-a734-1c93e8bac082-istiod-ca-cert" pod "762e23fe-a544-49c1-a734-1c93e8bac082" (UID: "762e23fe-a544-49c1-a734-1c93e8bac082"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:05:46.880429 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:46.879828    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/762e23fe-a544-49c1-a734-1c93e8bac082-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762e23fe-a544-49c1-a734-1c93e8bac082-workload-socket\") pod \"762e23fe-a544-49c1-a734-1c93e8bac082\" (UID: \"762e23fe-a544-49c1-a734-1c93e8bac082\") "
Oct 30 01:05:46.880429 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:46.880065    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762e23fe-a544-49c1-a734-1c93e8bac082-workload-certs" pod "762e23fe-a544-49c1-a734-1c93e8bac082" (UID: "762e23fe-a544-49c1-a734-1c93e8bac082"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:46.880429 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:46.880089    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762e23fe-a544-49c1-a734-1c93e8bac082-credential-socket" pod "762e23fe-a544-49c1-a734-1c93e8bac082" (UID: "762e23fe-a544-49c1-a734-1c93e8bac082"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:46.881415 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:46.880573    4393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/762e23fe-a544-49c1-a734-1c93e8bac082-workload-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:05:46.881415 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:46.880595    4393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/762e23fe-a544-49c1-a734-1c93e8bac082-credential-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:05:46.881415 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:46.880610    4393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/762e23fe-a544-49c1-a734-1c93e8bac082-istio-data\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:05:46.881415 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:46.880623    4393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/762e23fe-a544-49c1-a734-1c93e8bac082-istiod-ca-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:05:46.881415 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:46.881098    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762e23fe-a544-49c1-a734-1c93e8bac082-workload-socket" pod "762e23fe-a544-49c1-a734-1c93e8bac082" (UID: "762e23fe-a544-49c1-a734-1c93e8bac082"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:46.882692 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:46.882532    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762e23fe-a544-49c1-a734-1c93e8bac082-istio-envoy" pod "762e23fe-a544-49c1-a734-1c93e8bac082" (UID: "762e23fe-a544-49c1-a734-1c93e8bac082"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:05:46.882971 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:46.882943    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/762e23fe-a544-49c1-a734-1c93e8bac082-kube-api-access-sg779" pod "762e23fe-a544-49c1-a734-1c93e8bac082" (UID: "762e23fe-a544-49c1-a734-1c93e8bac082"). InnerVolumeSpecName "kube-api-access-sg779". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:46.883663 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:46.883641    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/762e23fe-a544-49c1-a734-1c93e8bac082-istio-token" pod "762e23fe-a544-49c1-a734-1c93e8bac082" (UID: "762e23fe-a544-49c1-a734-1c93e8bac082"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:46.883839 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:46.883819    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/762e23fe-a544-49c1-a734-1c93e8bac082-istio-podinfo" pod "762e23fe-a544-49c1-a734-1c93e8bac082" (UID: "762e23fe-a544-49c1-a734-1c93e8bac082"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:05:46.985124 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:46.984674    4393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/762e23fe-a544-49c1-a734-1c93e8bac082-istio-envoy\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:05:46.985124 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:46.984707    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/762e23fe-a544-49c1-a734-1c93e8bac082-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:05:46.985124 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:46.984720    4393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/762e23fe-a544-49c1-a734-1c93e8bac082-workload-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:05:46.985124 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:46.984733    4393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/762e23fe-a544-49c1-a734-1c93e8bac082-istio-podinfo\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:05:46.985124 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:46.984747    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg779\" (UniqueName: \"kubernetes.io/projected/762e23fe-a544-49c1-a734-1c93e8bac082-kube-api-access-sg779\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:05:46.998862 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:46.998826    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6"]
Oct 30 01:05:47.019839 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:47.019073    4393 generic.go:356] "Generic (PLEG): container finished" podID="762e23fe-a544-49c1-a734-1c93e8bac082" containerID="14266b28595a9771a400bc3f01c7907a46f2aed1779a72615e1620a7638d9706" exitCode=0
Oct 30 01:05:47.019839 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:47.019148    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw" event={"ID":"762e23fe-a544-49c1-a734-1c93e8bac082","Type":"ContainerDied","Data":"14266b28595a9771a400bc3f01c7907a46f2aed1779a72615e1620a7638d9706"}
Oct 30 01:05:47.019839 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:47.019212    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw" event={"ID":"762e23fe-a544-49c1-a734-1c93e8bac082","Type":"ContainerDied","Data":"54fd75452dd897ce8160c4413ca0b021b662572f22267a6dc2939dd6b1582066"}
Oct 30 01:05:47.019839 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:47.019246    4393 scope.go:122] "RemoveContainer" containerID="14266b28595a9771a400bc3f01c7907a46f2aed1779a72615e1620a7638d9706"
Oct 30 01:05:47.021177 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:47.021155    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"
Oct 30 01:05:47.036864 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:47.036845    4393 scope.go:122] "RemoveContainer" containerID="14266b28595a9771a400bc3f01c7907a46f2aed1779a72615e1620a7638d9706"
Oct 30 01:05:47.037183 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:05:47.037163    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14266b28595a9771a400bc3f01c7907a46f2aed1779a72615e1620a7638d9706\": container with ID starting with 14266b28595a9771a400bc3f01c7907a46f2aed1779a72615e1620a7638d9706 not found: ID does not exist" containerID="14266b28595a9771a400bc3f01c7907a46f2aed1779a72615e1620a7638d9706"
Oct 30 01:05:47.037235 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:47.037192    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14266b28595a9771a400bc3f01c7907a46f2aed1779a72615e1620a7638d9706"} err="failed to get container status \"14266b28595a9771a400bc3f01c7907a46f2aed1779a72615e1620a7638d9706\": rpc error: code = NotFound desc = could not find container \"14266b28595a9771a400bc3f01c7907a46f2aed1779a72615e1620a7638d9706\": container with ID starting with 14266b28595a9771a400bc3f01c7907a46f2aed1779a72615e1620a7638d9706 not found: ID does not exist"
Oct 30 01:05:47.194925 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:47.192128    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"]
Oct 30 01:05:47.214838 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:47.212695    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8d2a70-openshift-default-677474fcd7-899lw"]
Oct 30 01:05:47.728197 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:47.728164    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="762e23fe-a544-49c1-a734-1c93e8bac082" path="/var/lib/kubelet/pods/762e23fe-a544-49c1-a734-1c93e8bac082/volumes"
Oct 30 01:05:53.062678 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:53.057976    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9"]
Oct 30 01:05:53.062678 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:53.058901    4393 cpu_manager.go:414] "containerMap: removing container" podUID="98869821-d2ad-490b-b71b-ea9d17af6327" containerName="maas-api"
Oct 30 01:05:53.062678 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:53.058921    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98869821-d2ad-490b-b71b-ea9d17af6327" containerName="maas-api"
Oct 30 01:05:53.062678 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:53.058933    4393 cpu_manager.go:414] "containerMap: removing container" podUID="762e23fe-a544-49c1-a734-1c93e8bac082" containerName="istio-proxy"
Oct 30 01:05:53.062678 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:53.058939    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="762e23fe-a544-49c1-a734-1c93e8bac082" containerName="istio-proxy"
Oct 30 01:05:53.062678 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:53.059098    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="98869821-d2ad-490b-b71b-ea9d17af6327" containerName="maas-api"
Oct 30 01:05:53.062678 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:53.059106    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="762e23fe-a544-49c1-a734-1c93e8bac082" containerName="istio-proxy"
Oct 30 01:05:53.065833 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:53.065766    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9"
Oct 30 01:05:53.083916 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:53.083884    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9"]
Oct 30 01:05:53.088594 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:53.088569    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-cce513\"/\"openshift-service-ca.crt\""
Oct 30 01:05:53.088768 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:53.088578    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-cce513\"/\"default-dockercfg-pklnp\""
Oct 30 01:05:53.093572 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:53.092763    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-cce513\"/\"kube-root-ca.crt\""
Oct 30 01:05:53.093572 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:53.092869    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-cce513\"/\"rate-test-model-cce513-kserve-self-signed-certs\""
Oct 30 01:05:53.159410 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:53.159368    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80640906-b954-482d-bec9-4ff13cd07180-tmp-dir\") pod \"rate-test-model-cce513-kserve-5577645597-xgpn9\" (UID: \"80640906-b954-482d-bec9-4ff13cd07180\") " pod="ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9"
Oct 30 01:05:53.159410 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:53.159411    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80640906-b954-482d-bec9-4ff13cd07180-home\") pod \"rate-test-model-cce513-kserve-5577645597-xgpn9\" (UID: \"80640906-b954-482d-bec9-4ff13cd07180\") " pod="ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9"
Oct 30 01:05:53.159658 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:53.159554    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80640906-b954-482d-bec9-4ff13cd07180-model-cache\") pod \"rate-test-model-cce513-kserve-5577645597-xgpn9\" (UID: \"80640906-b954-482d-bec9-4ff13cd07180\") " pod="ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9"
Oct 30 01:05:53.159759 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:53.159737    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6w85\" (UniqueName: \"kubernetes.io/projected/80640906-b954-482d-bec9-4ff13cd07180-kube-api-access-q6w85\") pod \"rate-test-model-cce513-kserve-5577645597-xgpn9\" (UID: \"80640906-b954-482d-bec9-4ff13cd07180\") " pod="ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9"
Oct 30 01:05:53.159884 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:53.159868    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80640906-b954-482d-bec9-4ff13cd07180-tls-certs\") pod \"rate-test-model-cce513-kserve-5577645597-xgpn9\" (UID: \"80640906-b954-482d-bec9-4ff13cd07180\") " pod="ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9"
Oct 30 01:05:53.159950 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:53.159932    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80640906-b954-482d-bec9-4ff13cd07180-dshm\") pod \"rate-test-model-cce513-kserve-5577645597-xgpn9\" (UID: \"80640906-b954-482d-bec9-4ff13cd07180\") " pod="ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9"
Oct 30 01:05:53.261660 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:53.261623    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80640906-b954-482d-bec9-4ff13cd07180-dshm\") pod \"rate-test-model-cce513-kserve-5577645597-xgpn9\" (UID: \"80640906-b954-482d-bec9-4ff13cd07180\") " pod="ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9"
Oct 30 01:05:53.261894 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:53.261727    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80640906-b954-482d-bec9-4ff13cd07180-tmp-dir\") pod \"rate-test-model-cce513-kserve-5577645597-xgpn9\" (UID: \"80640906-b954-482d-bec9-4ff13cd07180\") " pod="ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9"
Oct 30 01:05:53.261894 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:53.261744    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80640906-b954-482d-bec9-4ff13cd07180-home\") pod \"rate-test-model-cce513-kserve-5577645597-xgpn9\" (UID: \"80640906-b954-482d-bec9-4ff13cd07180\") " pod="ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9"
Oct 30 01:05:53.262022 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:53.261900    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80640906-b954-482d-bec9-4ff13cd07180-model-cache\") pod \"rate-test-model-cce513-kserve-5577645597-xgpn9\" (UID: \"80640906-b954-482d-bec9-4ff13cd07180\") " pod="ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9"
Oct 30 01:05:53.262081 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:53.262040    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6w85\" (UniqueName: \"kubernetes.io/projected/80640906-b954-482d-bec9-4ff13cd07180-kube-api-access-q6w85\") pod \"rate-test-model-cce513-kserve-5577645597-xgpn9\" (UID: \"80640906-b954-482d-bec9-4ff13cd07180\") " pod="ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9"
Oct 30 01:05:53.262142 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:53.262113    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80640906-b954-482d-bec9-4ff13cd07180-tls-certs\") pod \"rate-test-model-cce513-kserve-5577645597-xgpn9\" (UID: \"80640906-b954-482d-bec9-4ff13cd07180\") " pod="ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9"
Oct 30 01:05:53.262213 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:53.262151    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80640906-b954-482d-bec9-4ff13cd07180-home\") pod \"rate-test-model-cce513-kserve-5577645597-xgpn9\" (UID: \"80640906-b954-482d-bec9-4ff13cd07180\") " pod="ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9"
Oct 30 01:05:53.262273 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:53.262228    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80640906-b954-482d-bec9-4ff13cd07180-model-cache\") pod \"rate-test-model-cce513-kserve-5577645597-xgpn9\" (UID: \"80640906-b954-482d-bec9-4ff13cd07180\") " pod="ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9"
Oct 30 01:05:53.262358 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:53.262336    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80640906-b954-482d-bec9-4ff13cd07180-tmp-dir\") pod \"rate-test-model-cce513-kserve-5577645597-xgpn9\" (UID: \"80640906-b954-482d-bec9-4ff13cd07180\") " pod="ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9"
Oct 30 01:05:53.264153 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:53.264123    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80640906-b954-482d-bec9-4ff13cd07180-dshm\") pod \"rate-test-model-cce513-kserve-5577645597-xgpn9\" (UID: \"80640906-b954-482d-bec9-4ff13cd07180\") " pod="ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9"
Oct 30 01:05:53.264634 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:53.264615    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80640906-b954-482d-bec9-4ff13cd07180-tls-certs\") pod \"rate-test-model-cce513-kserve-5577645597-xgpn9\" (UID: \"80640906-b954-482d-bec9-4ff13cd07180\") " pod="ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9"
Oct 30 01:05:53.310804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:53.310742    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6w85\" (UniqueName: \"kubernetes.io/projected/80640906-b954-482d-bec9-4ff13cd07180-kube-api-access-q6w85\") pod \"rate-test-model-cce513-kserve-5577645597-xgpn9\" (UID: \"80640906-b954-482d-bec9-4ff13cd07180\") " pod="ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9"
Oct 30 01:05:53.390847 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:53.385158    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9"
Oct 30 01:05:53.742806 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:53.742762    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9"]
Oct 30 01:05:53.743658 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:05:53.743629    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80640906_b954_482d_bec9_4ff13cd07180.slice/crio-e08888ed8611aebb2b4101ddf6d393bca4d12bf8254a759fa93f8195d5efaeaa WatchSource:0}: Error finding container e08888ed8611aebb2b4101ddf6d393bca4d12bf8254a759fa93f8195d5efaeaa: Status 404 returned error can't find the container with id e08888ed8611aebb2b4101ddf6d393bca4d12bf8254a759fa93f8195d5efaeaa
Oct 30 01:05:54.107566 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:54.107477    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9" event={"ID":"80640906-b954-482d-bec9-4ff13cd07180","Type":"ContainerStarted","Data":"b20c60eaa0c06ecb158788eab3299b093710fe807d05dbd6eae9bf00240a57f9"}
Oct 30 01:05:54.107566 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:54.107529    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9" event={"ID":"80640906-b954-482d-bec9-4ff13cd07180","Type":"ContainerStarted","Data":"e08888ed8611aebb2b4101ddf6d393bca4d12bf8254a759fa93f8195d5efaeaa"}
Oct 30 01:05:54.153699 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:54.153633    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9" podStartSLOduration=1.1536149390000001 podStartE2EDuration="1.153614939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:05:54.149564747 +0000 UTC m=+2044.537241137" watchObservedRunningTime="2026-10-30 01:05:54.153614939 +0000 UTC m=+2044.541291327"
Oct 30 01:05:54.389046 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:54.389010    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9"
Oct 30 01:05:56.512229 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:05:56.512193    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:05:57.353155 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:57.353113    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg"]
Oct 30 01:05:57.353434 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:57.353406    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg" podUID="84c761db-2eb0-445d-b6e6-5c22476c834a" containerName="maas-api" containerID="cri-o://1936a931e8a5df6c77370fb7bbe38df1eb606bcb99851f840f9d6addcedaa398" gracePeriod=30
Oct 30 01:05:57.708043 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:57.708011    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg"
Oct 30 01:05:57.910804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:57.902836    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84c761db-2eb0-445d-b6e6-5c22476c834a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84c761db-2eb0-445d-b6e6-5c22476c834a-maas-api-tls\") pod \"84c761db-2eb0-445d-b6e6-5c22476c834a\" (UID: \"84c761db-2eb0-445d-b6e6-5c22476c834a\") "
Oct 30 01:05:57.910804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:57.902966    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84c761db-2eb0-445d-b6e6-5c22476c834a-kube-api-access-lg87w\" (UniqueName: \"kubernetes.io/projected/84c761db-2eb0-445d-b6e6-5c22476c834a-kube-api-access-lg87w\") pod \"84c761db-2eb0-445d-b6e6-5c22476c834a\" (UID: \"84c761db-2eb0-445d-b6e6-5c22476c834a\") "
Oct 30 01:05:57.910804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:57.903042    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84c761db-2eb0-445d-b6e6-5c22476c834a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84c761db-2eb0-445d-b6e6-5c22476c834a-openshift-service-ca\") pod \"84c761db-2eb0-445d-b6e6-5c22476c834a\" (UID: \"84c761db-2eb0-445d-b6e6-5c22476c834a\") "
Oct 30 01:05:57.910804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:57.903218    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84c761db-2eb0-445d-b6e6-5c22476c834a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/84c761db-2eb0-445d-b6e6-5c22476c834a-metrics-certs\") pod \"84c761db-2eb0-445d-b6e6-5c22476c834a\" (UID: \"84c761db-2eb0-445d-b6e6-5c22476c834a\") "
Oct 30 01:05:58.018896 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:57.963122    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84c761db-2eb0-445d-b6e6-5c22476c834a-openshift-service-ca" pod "84c761db-2eb0-445d-b6e6-5c22476c834a" (UID: "84c761db-2eb0-445d-b6e6-5c22476c834a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:05:58.018896 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:58.002394    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84c761db-2eb0-445d-b6e6-5c22476c834a-maas-api-tls" pod "84c761db-2eb0-445d-b6e6-5c22476c834a" (UID: "84c761db-2eb0-445d-b6e6-5c22476c834a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:05:58.035958 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:58.031544    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84c761db-2eb0-445d-b6e6-5c22476c834a-metrics-certs" pod "84c761db-2eb0-445d-b6e6-5c22476c834a" (UID: "84c761db-2eb0-445d-b6e6-5c22476c834a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:05:58.035958 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:58.033424    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c761db-2eb0-445d-b6e6-5c22476c834a-kube-api-access-lg87w" pod "84c761db-2eb0-445d-b6e6-5c22476c834a" (UID: "84c761db-2eb0-445d-b6e6-5c22476c834a"). InnerVolumeSpecName "kube-api-access-lg87w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:05:58.035958 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:58.033886    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84c761db-2eb0-445d-b6e6-5c22476c834a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/84c761db-2eb0-445d-b6e6-5c22476c834a-metrics-certs\") pod \"84c761db-2eb0-445d-b6e6-5c22476c834a\" (UID: \"84c761db-2eb0-445d-b6e6-5c22476c834a\") "
Oct 30 01:05:58.035958 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:05:58.034027    4393 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/84c761db-2eb0-445d-b6e6-5c22476c834a/volumes/kubernetes.io~secret/metrics-certs
Oct 30 01:05:58.035958 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:58.034040    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84c761db-2eb0-445d-b6e6-5c22476c834a-metrics-certs" pod "84c761db-2eb0-445d-b6e6-5c22476c834a" (UID: "84c761db-2eb0-445d-b6e6-5c22476c834a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:05:58.035958 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:58.034982    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84c761db-2eb0-445d-b6e6-5c22476c834a-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:05:58.035958 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:58.035018    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84c761db-2eb0-445d-b6e6-5c22476c834a-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:05:58.035958 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:58.035035    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg87w\" (UniqueName: \"kubernetes.io/projected/84c761db-2eb0-445d-b6e6-5c22476c834a-kube-api-access-lg87w\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:05:58.035958 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:58.035052    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84c761db-2eb0-445d-b6e6-5c22476c834a-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:05:58.141174 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:58.141141    4393 generic.go:356] "Generic (PLEG): container finished" podID="84c761db-2eb0-445d-b6e6-5c22476c834a" containerID="1936a931e8a5df6c77370fb7bbe38df1eb606bcb99851f840f9d6addcedaa398" exitCode=0
Oct 30 01:05:58.141360 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:58.141190    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg"
Oct 30 01:05:58.141360 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:58.141226    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg" event={"ID":"84c761db-2eb0-445d-b6e6-5c22476c834a","Type":"ContainerDied","Data":"1936a931e8a5df6c77370fb7bbe38df1eb606bcb99851f840f9d6addcedaa398"}
Oct 30 01:05:58.141360 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:58.141284    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg" event={"ID":"84c761db-2eb0-445d-b6e6-5c22476c834a","Type":"ContainerDied","Data":"38f67ea97bb4a56e8c1ddfb8bdf15f332ce2552f503fe8d85daae6289b780e30"}
Oct 30 01:05:58.141360 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:58.141314    4393 scope.go:122] "RemoveContainer" containerID="1936a931e8a5df6c77370fb7bbe38df1eb606bcb99851f840f9d6addcedaa398"
Oct 30 01:05:58.158908 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:58.158882    4393 scope.go:122] "RemoveContainer" containerID="1936a931e8a5df6c77370fb7bbe38df1eb606bcb99851f840f9d6addcedaa398"
Oct 30 01:05:58.161419 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:05:58.161385    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1936a931e8a5df6c77370fb7bbe38df1eb606bcb99851f840f9d6addcedaa398\": container with ID starting with 1936a931e8a5df6c77370fb7bbe38df1eb606bcb99851f840f9d6addcedaa398 not found: ID does not exist" containerID="1936a931e8a5df6c77370fb7bbe38df1eb606bcb99851f840f9d6addcedaa398"
Oct 30 01:05:58.161543 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:58.161434    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1936a931e8a5df6c77370fb7bbe38df1eb606bcb99851f840f9d6addcedaa398"} err="failed to get container status \"1936a931e8a5df6c77370fb7bbe38df1eb606bcb99851f840f9d6addcedaa398\": rpc error: code = NotFound desc = could not find container \"1936a931e8a5df6c77370fb7bbe38df1eb606bcb99851f840f9d6addcedaa398\": container with ID starting with 1936a931e8a5df6c77370fb7bbe38df1eb606bcb99851f840f9d6addcedaa398 not found: ID does not exist"
Oct 30 01:05:58.193638 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:58.193605    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg"]
Oct 30 01:05:58.205187 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:58.205154    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-51b95e-5ff7bbd948-jkzbg"]
Oct 30 01:05:59.726530 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:05:59.726496    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84c761db-2eb0-445d-b6e6-5c22476c834a" path="/var/lib/kubelet/pods/84c761db-2eb0-445d-b6e6-5c22476c834a/volumes"
Oct 30 01:06:01.923224 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:01.923186    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"]
Oct 30 01:06:01.923673 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:01.923441    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6" podUID="dc80bac8-5fa1-4c5e-bdb1-73225ad3181c" containerName="istio-proxy" containerID="cri-o://118235a0a3a884f51100eb415119c01c80bc67f64d03eb58c146604fcad69cab" gracePeriod=30
Oct 30 01:06:01.935127 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:01.935095    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"]
Oct 30 01:06:01.935339 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:01.935316    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv" podUID="a5b4591e-8713-40c5-b33c-e6505356ee29" containerName="istio-proxy" containerID="cri-o://0463d565486a83116c8f7d135d7ac13b1d0cac60f488c2b116caeddda4a5959a" gracePeriod=30
Oct 30 01:06:02.028748 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:02.028700    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6" podUID="5985d486-ea0a-4703-9cf9-c2ba1c55f7ba" containerName="main" containerID="cri-o://c70681b9ce3da8784e24537f6cccb357583f869f055b4e77551551319bd49518" gracePeriod=45
Oct 30 01:06:02.222379 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:02.222282    4393 generic.go:356] "Generic (PLEG): container finished" podID="5985d486-ea0a-4703-9cf9-c2ba1c55f7ba" containerID="c70681b9ce3da8784e24537f6cccb357583f869f055b4e77551551319bd49518" exitCode=0
Oct 30 01:06:02.222379 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:02.222340    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6" event={"ID":"5985d486-ea0a-4703-9cf9-c2ba1c55f7ba","Type":"ContainerDied","Data":"c70681b9ce3da8784e24537f6cccb357583f869f055b4e77551551319bd49518"}
Oct 30 01:06:02.410739 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:02.410705    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6"
Oct 30 01:06:02.543305 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:02.543208    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-model-cache\") pod \"5985d486-ea0a-4703-9cf9-c2ba1c55f7ba\" (UID: \"5985d486-ea0a-4703-9cf9-c2ba1c55f7ba\") "
Oct 30 01:06:02.543484 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:02.543303    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-dshm\") pod \"5985d486-ea0a-4703-9cf9-c2ba1c55f7ba\" (UID: \"5985d486-ea0a-4703-9cf9-c2ba1c55f7ba\") "
Oct 30 01:06:02.543484 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:02.543404    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-kube-api-access-g4md4\" (UniqueName: \"kubernetes.io/projected/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-kube-api-access-g4md4\") pod \"5985d486-ea0a-4703-9cf9-c2ba1c55f7ba\" (UID: \"5985d486-ea0a-4703-9cf9-c2ba1c55f7ba\") "
Oct 30 01:06:02.543602 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:02.543535    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-model-cache" pod "5985d486-ea0a-4703-9cf9-c2ba1c55f7ba" (UID: "5985d486-ea0a-4703-9cf9-c2ba1c55f7ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:02.543602 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:02.543574    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-tls-certs\" (UniqueName: \"kubernetes.io/secret/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-tls-certs\") pod \"5985d486-ea0a-4703-9cf9-c2ba1c55f7ba\" (UID: \"5985d486-ea0a-4703-9cf9-c2ba1c55f7ba\") "
Oct 30 01:06:02.543679 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:02.543654    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-tmp-dir\") pod \"5985d486-ea0a-4703-9cf9-c2ba1c55f7ba\" (UID: \"5985d486-ea0a-4703-9cf9-c2ba1c55f7ba\") "
Oct 30 01:06:02.543766 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:02.543751    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-home\" (UniqueName: \"kubernetes.io/empty-dir/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-home\") pod \"5985d486-ea0a-4703-9cf9-c2ba1c55f7ba\" (UID: \"5985d486-ea0a-4703-9cf9-c2ba1c55f7ba\") "
Oct 30 01:06:02.543982 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:02.543957    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-tmp-dir" pod "5985d486-ea0a-4703-9cf9-c2ba1c55f7ba" (UID: "5985d486-ea0a-4703-9cf9-c2ba1c55f7ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:02.544077 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:02.544053    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-home" pod "5985d486-ea0a-4703-9cf9-c2ba1c55f7ba" (UID: "5985d486-ea0a-4703-9cf9-c2ba1c55f7ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:02.544395 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:02.544377    4393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-tmp-dir\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:02.544463 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:02.544396    4393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-home\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:02.544463 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:02.544405    4393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-model-cache\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:02.545938 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:02.545902    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-dshm" pod "5985d486-ea0a-4703-9cf9-c2ba1c55f7ba" (UID: "5985d486-ea0a-4703-9cf9-c2ba1c55f7ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:02.546063 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:02.545967    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-kube-api-access-g4md4" pod "5985d486-ea0a-4703-9cf9-c2ba1c55f7ba" (UID: "5985d486-ea0a-4703-9cf9-c2ba1c55f7ba"). InnerVolumeSpecName "kube-api-access-g4md4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:02.546063 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:02.545984    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-tls-certs" pod "5985d486-ea0a-4703-9cf9-c2ba1c55f7ba" (UID: "5985d486-ea0a-4703-9cf9-c2ba1c55f7ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:06:02.646201 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:02.646160    4393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-tls-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:02.646201 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:02.646200    4393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-dshm\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:02.646466 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:02.646214    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4md4\" (UniqueName: \"kubernetes.io/projected/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba-kube-api-access-g4md4\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:02.930748 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:02.930715    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7"]
Oct 30 01:06:02.932358 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:02.932320    4393 cpu_manager.go:414] "containerMap: removing container" podUID="5985d486-ea0a-4703-9cf9-c2ba1c55f7ba" containerName="main"
Oct 30 01:06:02.932358 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:02.932345    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5985d486-ea0a-4703-9cf9-c2ba1c55f7ba" containerName="main"
Oct 30 01:06:02.932358 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:02.932360    4393 cpu_manager.go:414] "containerMap: removing container" podUID="84c761db-2eb0-445d-b6e6-5c22476c834a" containerName="maas-api"
Oct 30 01:06:02.932570 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:02.932366    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84c761db-2eb0-445d-b6e6-5c22476c834a" containerName="maas-api"
Oct 30 01:06:02.932570 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:02.932501    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="5985d486-ea0a-4703-9cf9-c2ba1c55f7ba" containerName="main"
Oct 30 01:06:02.932570 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:02.932515    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="84c761db-2eb0-445d-b6e6-5c22476c834a" containerName="maas-api"
Oct 30 01:06:02.936872 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:02.936849    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7"
Oct 30 01:06:02.954301 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:02.954255    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7"]
Oct 30 01:06:02.964858 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:02.960234    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-11f521\"/\"openshift-service-ca.crt\""
Oct 30 01:06:02.964858 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:02.960340    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-11f521\"/\"sub-test-model-11f521-kserve-self-signed-certs\""
Oct 30 01:06:02.964858 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:02.960402    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-11f521\"/\"default-dockercfg-kwnkx\""
Oct 30 01:06:02.964858 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:02.960441    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-11f521\"/\"kube-root-ca.crt\""
Oct 30 01:06:03.052508 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:03.052473    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bef7646-518d-4735-bee6-b1f804d5ea8e-dshm\") pod \"sub-test-model-11f521-kserve-57cd86885c-nkgt7\" (UID: \"8bef7646-518d-4735-bee6-b1f804d5ea8e\") " pod="ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7"
Oct 30 01:06:03.052508 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:03.052512    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bef7646-518d-4735-bee6-b1f804d5ea8e-tmp-dir\") pod \"sub-test-model-11f521-kserve-57cd86885c-nkgt7\" (UID: \"8bef7646-518d-4735-bee6-b1f804d5ea8e\") " pod="ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7"
Oct 30 01:06:03.052720 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:03.052537    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bef7646-518d-4735-bee6-b1f804d5ea8e-model-cache\") pod \"sub-test-model-11f521-kserve-57cd86885c-nkgt7\" (UID: \"8bef7646-518d-4735-bee6-b1f804d5ea8e\") " pod="ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7"
Oct 30 01:06:03.052720 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:03.052604    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bef7646-518d-4735-bee6-b1f804d5ea8e-home\") pod \"sub-test-model-11f521-kserve-57cd86885c-nkgt7\" (UID: \"8bef7646-518d-4735-bee6-b1f804d5ea8e\") " pod="ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7"
Oct 30 01:06:03.052720 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:03.052683    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bef7646-518d-4735-bee6-b1f804d5ea8e-tls-certs\") pod \"sub-test-model-11f521-kserve-57cd86885c-nkgt7\" (UID: \"8bef7646-518d-4735-bee6-b1f804d5ea8e\") " pod="ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7"
Oct 30 01:06:03.052857 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:03.052740    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f7l6\" (UniqueName: \"kubernetes.io/projected/8bef7646-518d-4735-bee6-b1f804d5ea8e-kube-api-access-9f7l6\") pod \"sub-test-model-11f521-kserve-57cd86885c-nkgt7\" (UID: \"8bef7646-518d-4735-bee6-b1f804d5ea8e\") " pod="ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7"
Oct 30 01:06:03.153896 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:03.153865    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bef7646-518d-4735-bee6-b1f804d5ea8e-dshm\") pod \"sub-test-model-11f521-kserve-57cd86885c-nkgt7\" (UID: \"8bef7646-518d-4735-bee6-b1f804d5ea8e\") " pod="ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7"
Oct 30 01:06:03.153896 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:03.153900    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bef7646-518d-4735-bee6-b1f804d5ea8e-tmp-dir\") pod \"sub-test-model-11f521-kserve-57cd86885c-nkgt7\" (UID: \"8bef7646-518d-4735-bee6-b1f804d5ea8e\") " pod="ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7"
Oct 30 01:06:03.154129 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:03.153924    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bef7646-518d-4735-bee6-b1f804d5ea8e-model-cache\") pod \"sub-test-model-11f521-kserve-57cd86885c-nkgt7\" (UID: \"8bef7646-518d-4735-bee6-b1f804d5ea8e\") " pod="ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7"
Oct 30 01:06:03.154129 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:03.154042    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bef7646-518d-4735-bee6-b1f804d5ea8e-home\") pod \"sub-test-model-11f521-kserve-57cd86885c-nkgt7\" (UID: \"8bef7646-518d-4735-bee6-b1f804d5ea8e\") " pod="ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7"
Oct 30 01:06:03.154129 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:03.154114    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bef7646-518d-4735-bee6-b1f804d5ea8e-tls-certs\") pod \"sub-test-model-11f521-kserve-57cd86885c-nkgt7\" (UID: \"8bef7646-518d-4735-bee6-b1f804d5ea8e\") " pod="ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7"
Oct 30 01:06:03.154273 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:03.154181    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f7l6\" (UniqueName: \"kubernetes.io/projected/8bef7646-518d-4735-bee6-b1f804d5ea8e-kube-api-access-9f7l6\") pod \"sub-test-model-11f521-kserve-57cd86885c-nkgt7\" (UID: \"8bef7646-518d-4735-bee6-b1f804d5ea8e\") " pod="ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7"
Oct 30 01:06:03.154339 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:03.154290    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bef7646-518d-4735-bee6-b1f804d5ea8e-tmp-dir\") pod \"sub-test-model-11f521-kserve-57cd86885c-nkgt7\" (UID: \"8bef7646-518d-4735-bee6-b1f804d5ea8e\") " pod="ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7"
Oct 30 01:06:03.154394 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:03.154337    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bef7646-518d-4735-bee6-b1f804d5ea8e-model-cache\") pod \"sub-test-model-11f521-kserve-57cd86885c-nkgt7\" (UID: \"8bef7646-518d-4735-bee6-b1f804d5ea8e\") " pod="ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7"
Oct 30 01:06:03.154468 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:03.154449    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bef7646-518d-4735-bee6-b1f804d5ea8e-home\") pod \"sub-test-model-11f521-kserve-57cd86885c-nkgt7\" (UID: \"8bef7646-518d-4735-bee6-b1f804d5ea8e\") " pod="ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7"
Oct 30 01:06:03.156546 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:03.156513    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bef7646-518d-4735-bee6-b1f804d5ea8e-dshm\") pod \"sub-test-model-11f521-kserve-57cd86885c-nkgt7\" (UID: \"8bef7646-518d-4735-bee6-b1f804d5ea8e\") " pod="ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7"
Oct 30 01:06:03.156709 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:03.156683    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bef7646-518d-4735-bee6-b1f804d5ea8e-tls-certs\") pod \"sub-test-model-11f521-kserve-57cd86885c-nkgt7\" (UID: \"8bef7646-518d-4735-bee6-b1f804d5ea8e\") " pod="ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7"
Oct 30 01:06:03.176894 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:03.176866    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f7l6\" (UniqueName: \"kubernetes.io/projected/8bef7646-518d-4735-bee6-b1f804d5ea8e-kube-api-access-9f7l6\") pod \"sub-test-model-11f521-kserve-57cd86885c-nkgt7\" (UID: \"8bef7646-518d-4735-bee6-b1f804d5ea8e\") " pod="ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7"
Oct 30 01:06:03.292297 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:03.292203    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7"
Oct 30 01:06:03.331290 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:03.331242    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6" event={"ID":"5985d486-ea0a-4703-9cf9-c2ba1c55f7ba","Type":"ContainerDied","Data":"e26e1a1b14e70847ab56c5c19982b6becf420da5a757c775641463cdb16b17dd"}
Oct 30 01:06:03.331492 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:03.331305    4393 scope.go:122] "RemoveContainer" containerID="c70681b9ce3da8784e24537f6cccb357583f869f055b4e77551551319bd49518"
Oct 30 01:06:03.331558 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:03.331506    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6"
Oct 30 01:06:03.388973 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:03.388946    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9"
Oct 30 01:06:03.406340 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:03.406309    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9"
Oct 30 01:06:03.427220 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:03.421719    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6"]
Oct 30 01:06:03.428585 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:03.428555    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-b1f6ed/test-model-b1f6ed-kserve-5d547ddd7-whgr6"]
Oct 30 01:06:03.675608 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:06:03.675575    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bef7646_518d_4735_bee6_b1f804d5ea8e.slice/crio-708ffcf160ae8efc9c1838fe969ac974504664ccd420f399c6e455bf3fbbc7f8 WatchSource:0}: Error finding container 708ffcf160ae8efc9c1838fe969ac974504664ccd420f399c6e455bf3fbbc7f8: Status 404 returned error can't find the container with id 708ffcf160ae8efc9c1838fe969ac974504664ccd420f399c6e455bf3fbbc7f8
Oct 30 01:06:03.675762 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:03.675742    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7"]
Oct 30 01:06:03.729013 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:03.728987    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5985d486-ea0a-4703-9cf9-c2ba1c55f7ba" path="/var/lib/kubelet/pods/5985d486-ea0a-4703-9cf9-c2ba1c55f7ba/volumes"
Oct 30 01:06:04.342125 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:04.342083    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7" event={"ID":"8bef7646-518d-4735-bee6-b1f804d5ea8e","Type":"ContainerStarted","Data":"ebd8c14f2c520f2d44edaf3de2ce15b1407917cd1ab75c80bff55083c6783576"}
Oct 30 01:06:04.342643 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:04.342134    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7" event={"ID":"8bef7646-518d-4735-bee6-b1f804d5ea8e","Type":"ContainerStarted","Data":"708ffcf160ae8efc9c1838fe969ac974504664ccd420f399c6e455bf3fbbc7f8"}
Oct 30 01:06:04.353849 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:04.353815    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9"
Oct 30 01:06:04.384808 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:04.384730    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7" podStartSLOduration=2.384715537 podStartE2EDuration="2.384715537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:04.381649166 +0000 UTC m=+2054.769325556" watchObservedRunningTime="2026-10-30 01:06:04.384715537 +0000 UTC m=+2054.772391972"
Oct 30 01:06:05.296688 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:05.296653    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7"
Oct 30 01:06:07.253847 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.253816    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"
Oct 30 01:06:07.292740 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.292713    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"
Oct 30 01:06:07.358489 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.356900    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5b4591e-8713-40c5-b33c-e6505356ee29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5b4591e-8713-40c5-b33c-e6505356ee29-istio-data\") pod \"a5b4591e-8713-40c5-b33c-e6505356ee29\" (UID: \"a5b4591e-8713-40c5-b33c-e6505356ee29\") "
Oct 30 01:06:07.358489 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.357022    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5b4591e-8713-40c5-b33c-e6505356ee29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b4591e-8713-40c5-b33c-e6505356ee29-workload-socket\") pod \"a5b4591e-8713-40c5-b33c-e6505356ee29\" (UID: \"a5b4591e-8713-40c5-b33c-e6505356ee29\") "
Oct 30 01:06:07.358489 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.357308    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b4591e-8713-40c5-b33c-e6505356ee29-istio-data" pod "a5b4591e-8713-40c5-b33c-e6505356ee29" (UID: "a5b4591e-8713-40c5-b33c-e6505356ee29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:07.358489 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.357644    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b4591e-8713-40c5-b33c-e6505356ee29-workload-socket" pod "a5b4591e-8713-40c5-b33c-e6505356ee29" (UID: "a5b4591e-8713-40c5-b33c-e6505356ee29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:07.358489 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.357743    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5b4591e-8713-40c5-b33c-e6505356ee29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5b4591e-8713-40c5-b33c-e6505356ee29-istio-envoy\") pod \"a5b4591e-8713-40c5-b33c-e6505356ee29\" (UID: \"a5b4591e-8713-40c5-b33c-e6505356ee29\") "
Oct 30 01:06:07.358489 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.357802    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-istio-data\") pod \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\" (UID: \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\") "
Oct 30 01:06:07.358489 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.357869    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-istio-envoy\") pod \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\" (UID: \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\") "
Oct 30 01:06:07.358489 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.357911    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-credential-socket\") pod \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\" (UID: \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\") "
Oct 30 01:06:07.358489 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.357951    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5b4591e-8713-40c5-b33c-e6505356ee29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b4591e-8713-40c5-b33c-e6505356ee29-credential-socket\") pod \"a5b4591e-8713-40c5-b33c-e6505356ee29\" (UID: \"a5b4591e-8713-40c5-b33c-e6505356ee29\") "
Oct 30 01:06:07.358489 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.357978    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-kube-api-access-756zh\" (UniqueName: \"kubernetes.io/projected/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-kube-api-access-756zh\") pod \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\" (UID: \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\") "
Oct 30 01:06:07.358489 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.358056    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-workload-socket\") pod \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\" (UID: \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\") "
Oct 30 01:06:07.358489 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.358093    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5b4591e-8713-40c5-b33c-e6505356ee29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5b4591e-8713-40c5-b33c-e6505356ee29-istiod-ca-cert\") pod \"a5b4591e-8713-40c5-b33c-e6505356ee29\" (UID: \"a5b4591e-8713-40c5-b33c-e6505356ee29\") "
Oct 30 01:06:07.358489 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.358141    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5b4591e-8713-40c5-b33c-e6505356ee29-kube-api-access-mxfkf\" (UniqueName: \"kubernetes.io/projected/a5b4591e-8713-40c5-b33c-e6505356ee29-kube-api-access-mxfkf\") pod \"a5b4591e-8713-40c5-b33c-e6505356ee29\" (UID: \"a5b4591e-8713-40c5-b33c-e6505356ee29\") "
Oct 30 01:06:07.358489 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.358239    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5b4591e-8713-40c5-b33c-e6505356ee29-istio-token\" (UniqueName: \"kubernetes.io/projected/a5b4591e-8713-40c5-b33c-e6505356ee29-istio-token\") pod \"a5b4591e-8713-40c5-b33c-e6505356ee29\" (UID: \"a5b4591e-8713-40c5-b33c-e6505356ee29\") "
Oct 30 01:06:07.358489 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.358324    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5b4591e-8713-40c5-b33c-e6505356ee29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5b4591e-8713-40c5-b33c-e6505356ee29-workload-certs\") pod \"a5b4591e-8713-40c5-b33c-e6505356ee29\" (UID: \"a5b4591e-8713-40c5-b33c-e6505356ee29\") "
Oct 30 01:06:07.358489 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.358398    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-istiod-ca-cert\") pod \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\" (UID: \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\") "
Oct 30 01:06:07.360176 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.358655    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b4591e-8713-40c5-b33c-e6505356ee29-credential-socket" pod "a5b4591e-8713-40c5-b33c-e6505356ee29" (UID: "a5b4591e-8713-40c5-b33c-e6505356ee29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:07.360176 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.359020    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5b4591e-8713-40c5-b33c-e6505356ee29-istiod-ca-cert" pod "a5b4591e-8713-40c5-b33c-e6505356ee29" (UID: "a5b4591e-8713-40c5-b33c-e6505356ee29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:06:07.360176 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.359357    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-credential-socket" pod "dc80bac8-5fa1-4c5e-bdb1-73225ad3181c" (UID: "dc80bac8-5fa1-4c5e-bdb1-73225ad3181c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:07.360176 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.359525    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-istio-podinfo\") pod \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\" (UID: \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\") "
Oct 30 01:06:07.360176 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.359580    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-workload-certs\") pod \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\" (UID: \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\") "
Oct 30 01:06:07.360176 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.359608    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-istio-token\" (UniqueName: \"kubernetes.io/projected/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-istio-token\") pod \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\" (UID: \"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c\") "
Oct 30 01:06:07.360176 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.359640    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5b4591e-8713-40c5-b33c-e6505356ee29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5b4591e-8713-40c5-b33c-e6505356ee29-istio-podinfo\") pod \"a5b4591e-8713-40c5-b33c-e6505356ee29\" (UID: \"a5b4591e-8713-40c5-b33c-e6505356ee29\") "
Oct 30 01:06:07.360176 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.359679    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-workload-socket" pod "dc80bac8-5fa1-4c5e-bdb1-73225ad3181c" (UID: "dc80bac8-5fa1-4c5e-bdb1-73225ad3181c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:07.360176 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.359932    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-istio-data" pod "dc80bac8-5fa1-4c5e-bdb1-73225ad3181c" (UID: "dc80bac8-5fa1-4c5e-bdb1-73225ad3181c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:07.360816 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.360506    4393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b4591e-8713-40c5-b33c-e6505356ee29-credential-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:07.360816 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.360530    4393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-workload-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:07.360816 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.360548    4393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5b4591e-8713-40c5-b33c-e6505356ee29-istiod-ca-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:07.360816 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.360561    4393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5b4591e-8713-40c5-b33c-e6505356ee29-istio-data\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:07.360816 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.360575    4393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b4591e-8713-40c5-b33c-e6505356ee29-workload-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:07.360816 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.360588    4393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-istio-data\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:07.360816 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.360602    4393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-credential-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:07.368162 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.368125    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5b4591e-8713-40c5-b33c-e6505356ee29-kube-api-access-mxfkf" pod "a5b4591e-8713-40c5-b33c-e6505356ee29" (UID: "a5b4591e-8713-40c5-b33c-e6505356ee29"). InnerVolumeSpecName "kube-api-access-mxfkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:07.368308 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.368272    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-workload-certs" pod "dc80bac8-5fa1-4c5e-bdb1-73225ad3181c" (UID: "dc80bac8-5fa1-4c5e-bdb1-73225ad3181c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:07.368515 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.368482    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b4591e-8713-40c5-b33c-e6505356ee29-workload-certs" pod "a5b4591e-8713-40c5-b33c-e6505356ee29" (UID: "a5b4591e-8713-40c5-b33c-e6505356ee29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:07.368643 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.368618    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-istiod-ca-cert" pod "dc80bac8-5fa1-4c5e-bdb1-73225ad3181c" (UID: "dc80bac8-5fa1-4c5e-bdb1-73225ad3181c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:06:07.370622 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.370572    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-istio-token" pod "dc80bac8-5fa1-4c5e-bdb1-73225ad3181c" (UID: "dc80bac8-5fa1-4c5e-bdb1-73225ad3181c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:07.371857 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.371831    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-istio-podinfo" pod "dc80bac8-5fa1-4c5e-bdb1-73225ad3181c" (UID: "dc80bac8-5fa1-4c5e-bdb1-73225ad3181c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:06:07.372321 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.372101    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b4591e-8713-40c5-b33c-e6505356ee29-istio-envoy" pod "a5b4591e-8713-40c5-b33c-e6505356ee29" (UID: "a5b4591e-8713-40c5-b33c-e6505356ee29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:07.373184 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.373106    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5b4591e-8713-40c5-b33c-e6505356ee29-istio-token" pod "a5b4591e-8713-40c5-b33c-e6505356ee29" (UID: "a5b4591e-8713-40c5-b33c-e6505356ee29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:07.373184 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.373150    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-kube-api-access-756zh" pod "dc80bac8-5fa1-4c5e-bdb1-73225ad3181c" (UID: "dc80bac8-5fa1-4c5e-bdb1-73225ad3181c"). InnerVolumeSpecName "kube-api-access-756zh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:07.373805 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.373695    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5b4591e-8713-40c5-b33c-e6505356ee29-istio-podinfo" pod "a5b4591e-8713-40c5-b33c-e6505356ee29" (UID: "a5b4591e-8713-40c5-b33c-e6505356ee29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:06:07.374804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.374743    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-istio-envoy" pod "dc80bac8-5fa1-4c5e-bdb1-73225ad3181c" (UID: "dc80bac8-5fa1-4c5e-bdb1-73225ad3181c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:07.384016 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.383990    4393 generic.go:356] "Generic (PLEG): container finished" podID="a5b4591e-8713-40c5-b33c-e6505356ee29" containerID="0463d565486a83116c8f7d135d7ac13b1d0cac60f488c2b116caeddda4a5959a" exitCode=0
Oct 30 01:06:07.384154 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.384055    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"
Oct 30 01:06:07.384154 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.384080    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv" event={"ID":"a5b4591e-8713-40c5-b33c-e6505356ee29","Type":"ContainerDied","Data":"0463d565486a83116c8f7d135d7ac13b1d0cac60f488c2b116caeddda4a5959a"}
Oct 30 01:06:07.384154 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.384137    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv" event={"ID":"a5b4591e-8713-40c5-b33c-e6505356ee29","Type":"ContainerDied","Data":"91125411b9d28215ed8c80f0a8b6518c6b2ba2f8609f21ff007b5731f1fe4ab3"}
Oct 30 01:06:07.384327 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.384165    4393 scope.go:122] "RemoveContainer" containerID="0463d565486a83116c8f7d135d7ac13b1d0cac60f488c2b116caeddda4a5959a"
Oct 30 01:06:07.386581 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.386557    4393 generic.go:356] "Generic (PLEG): container finished" podID="dc80bac8-5fa1-4c5e-bdb1-73225ad3181c" containerID="118235a0a3a884f51100eb415119c01c80bc67f64d03eb58c146604fcad69cab" exitCode=0
Oct 30 01:06:07.386701 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.386651    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"
Oct 30 01:06:07.386701 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.386685    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6" event={"ID":"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c","Type":"ContainerDied","Data":"118235a0a3a884f51100eb415119c01c80bc67f64d03eb58c146604fcad69cab"}
Oct 30 01:06:07.386900 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.386732    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6" event={"ID":"dc80bac8-5fa1-4c5e-bdb1-73225ad3181c","Type":"ContainerDied","Data":"4cdc9180ae99aa0cf9684ec9484661796a64d4fdae54017deedcb0106f9587fe"}
Oct 30 01:06:07.409697 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.409676    4393 scope.go:122] "RemoveContainer" containerID="0463d565486a83116c8f7d135d7ac13b1d0cac60f488c2b116caeddda4a5959a"
Oct 30 01:06:07.410047 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:06:07.410026    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0463d565486a83116c8f7d135d7ac13b1d0cac60f488c2b116caeddda4a5959a\": container with ID starting with 0463d565486a83116c8f7d135d7ac13b1d0cac60f488c2b116caeddda4a5959a not found: ID does not exist" containerID="0463d565486a83116c8f7d135d7ac13b1d0cac60f488c2b116caeddda4a5959a"
Oct 30 01:06:07.410134 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.410064    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0463d565486a83116c8f7d135d7ac13b1d0cac60f488c2b116caeddda4a5959a"} err="failed to get container status \"0463d565486a83116c8f7d135d7ac13b1d0cac60f488c2b116caeddda4a5959a\": rpc error: code = NotFound desc = could not find container \"0463d565486a83116c8f7d135d7ac13b1d0cac60f488c2b116caeddda4a5959a\": container with ID starting with 0463d565486a83116c8f7d135d7ac13b1d0cac60f488c2b116caeddda4a5959a not found: ID does not exist"
Oct 30 01:06:07.410134 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.410089    4393 scope.go:122] "RemoveContainer" containerID="118235a0a3a884f51100eb415119c01c80bc67f64d03eb58c146604fcad69cab"
Oct 30 01:06:07.421606 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.421582    4393 scope.go:122] "RemoveContainer" containerID="118235a0a3a884f51100eb415119c01c80bc67f64d03eb58c146604fcad69cab"
Oct 30 01:06:07.421931 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:06:07.421908    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"118235a0a3a884f51100eb415119c01c80bc67f64d03eb58c146604fcad69cab\": container with ID starting with 118235a0a3a884f51100eb415119c01c80bc67f64d03eb58c146604fcad69cab not found: ID does not exist" containerID="118235a0a3a884f51100eb415119c01c80bc67f64d03eb58c146604fcad69cab"
Oct 30 01:06:07.422010 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.421940    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"118235a0a3a884f51100eb415119c01c80bc67f64d03eb58c146604fcad69cab"} err="failed to get container status \"118235a0a3a884f51100eb415119c01c80bc67f64d03eb58c146604fcad69cab\": rpc error: code = NotFound desc = could not find container \"118235a0a3a884f51100eb415119c01c80bc67f64d03eb58c146604fcad69cab\": container with ID starting with 118235a0a3a884f51100eb415119c01c80bc67f64d03eb58c146604fcad69cab not found: ID does not exist"
Oct 30 01:06:07.452669 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.452635    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"]
Oct 30 01:06:07.456933 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.456868    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-xx8n6"]
Oct 30 01:06:07.461818 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.461735    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5b4591e-8713-40c5-b33c-e6505356ee29-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:07.461818 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.461763    4393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5b4591e-8713-40c5-b33c-e6505356ee29-workload-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:07.461818 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.461794    4393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-istiod-ca-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:07.461818 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.461810    4393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-istio-podinfo\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:07.461818 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.461822    4393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-workload-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:07.462053 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.461833    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:07.462053 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.461845    4393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5b4591e-8713-40c5-b33c-e6505356ee29-istio-podinfo\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:07.462053 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.461856    4393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5b4591e-8713-40c5-b33c-e6505356ee29-istio-envoy\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:07.462053 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.461869    4393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-istio-envoy\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:07.462053 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.461880    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-756zh\" (UniqueName: \"kubernetes.io/projected/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c-kube-api-access-756zh\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:07.462053 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.461894    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxfkf\" (UniqueName: \"kubernetes.io/projected/a5b4591e-8713-40c5-b33c-e6505356ee29-kube-api-access-mxfkf\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:07.500281 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.500248    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"]
Oct 30 01:06:07.504904 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.504876    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-kfpcv"]
Oct 30 01:06:07.727121 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.727036    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5b4591e-8713-40c5-b33c-e6505356ee29" path="/var/lib/kubelet/pods/a5b4591e-8713-40c5-b33c-e6505356ee29/volumes"
Oct 30 01:06:07.727599 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:07.727584    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc80bac8-5fa1-4c5e-bdb1-73225ad3181c" path="/var/lib/kubelet/pods/dc80bac8-5fa1-4c5e-bdb1-73225ad3181c/volumes"
Oct 30 01:06:08.504955 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:08.504918    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"]
Oct 30 01:06:08.505412 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:08.505146    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x" podUID="2bc5c9f7-dbe6-4741-8e37-258a1f1ed604" containerName="istio-proxy" containerID="cri-o://e0ca147f03a24f57bae14caf8fca4797d04d0c40c7e3fbd85fb71f3ea5d8fee0" gracePeriod=30
Oct 30 01:06:08.625755 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:08.625719    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"]
Oct 30 01:06:08.625971 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:08.625940    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg" podUID="a8bdca6c-102d-4aaa-ab76-16a9123d6a74" containerName="istio-proxy" containerID="cri-o://434b71c4baabd19f38964a7311756922c8f8934b3b6c8beebe7df7898d3d01d8" gracePeriod=30
Oct 30 01:06:13.296714 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.296673    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7"
Oct 30 01:06:13.310152 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.310126    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7"
Oct 30 01:06:13.454376 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.454342    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7"
Oct 30 01:06:13.807241 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.807211    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"
Oct 30 01:06:13.874970 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.874862    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-credential-socket\") pod \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\" (UID: \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\") "
Oct 30 01:06:13.874970 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.874928    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-kube-api-access-5md4m\" (UniqueName: \"kubernetes.io/projected/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-kube-api-access-5md4m\") pod \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\" (UID: \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\") "
Oct 30 01:06:13.876389 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.875005    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-istio-data\") pod \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\" (UID: \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\") "
Oct 30 01:06:13.876389 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.875117    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-istiod-ca-cert\") pod \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\" (UID: \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\") "
Oct 30 01:06:13.876389 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.875148    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-istio-envoy\") pod \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\" (UID: \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\") "
Oct 30 01:06:13.876389 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.875213    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-istio-token\" (UniqueName: \"kubernetes.io/projected/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-istio-token\") pod \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\" (UID: \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\") "
Oct 30 01:06:13.876389 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.875233    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-credential-socket" pod "2bc5c9f7-dbe6-4741-8e37-258a1f1ed604" (UID: "2bc5c9f7-dbe6-4741-8e37-258a1f1ed604"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:13.876389 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.875317    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-workload-certs\") pod \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\" (UID: \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\") "
Oct 30 01:06:13.876389 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.875346    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-workload-socket\") pod \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\" (UID: \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\") "
Oct 30 01:06:13.876389 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.875397    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-istio-data" pod "2bc5c9f7-dbe6-4741-8e37-258a1f1ed604" (UID: "2bc5c9f7-dbe6-4741-8e37-258a1f1ed604"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:13.876389 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.875563    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-istio-podinfo\") pod \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\" (UID: \"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604\") "
Oct 30 01:06:13.876389 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.875628    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-istiod-ca-cert" pod "2bc5c9f7-dbe6-4741-8e37-258a1f1ed604" (UID: "2bc5c9f7-dbe6-4741-8e37-258a1f1ed604"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:06:13.876389 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.875972    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-workload-certs" pod "2bc5c9f7-dbe6-4741-8e37-258a1f1ed604" (UID: "2bc5c9f7-dbe6-4741-8e37-258a1f1ed604"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:13.876389 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.876213    4393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-credential-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:13.876389 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.876236    4393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-istio-data\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:13.876389 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.876250    4393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-istiod-ca-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:13.876389 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.876263    4393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-workload-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:13.877399 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.877094    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-workload-socket" pod "2bc5c9f7-dbe6-4741-8e37-258a1f1ed604" (UID: "2bc5c9f7-dbe6-4741-8e37-258a1f1ed604"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:13.879142 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.879113    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-kube-api-access-5md4m" pod "2bc5c9f7-dbe6-4741-8e37-258a1f1ed604" (UID: "2bc5c9f7-dbe6-4741-8e37-258a1f1ed604"). InnerVolumeSpecName "kube-api-access-5md4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:13.880059 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.880038    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-istio-podinfo" pod "2bc5c9f7-dbe6-4741-8e37-258a1f1ed604" (UID: "2bc5c9f7-dbe6-4741-8e37-258a1f1ed604"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:06:13.882952 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.882910    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-istio-token" pod "2bc5c9f7-dbe6-4741-8e37-258a1f1ed604" (UID: "2bc5c9f7-dbe6-4741-8e37-258a1f1ed604"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:13.883186 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.883153    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-istio-envoy" pod "2bc5c9f7-dbe6-4741-8e37-258a1f1ed604" (UID: "2bc5c9f7-dbe6-4741-8e37-258a1f1ed604"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:13.928865 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.928832    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"
Oct 30 01:06:13.980256 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.980214    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-workload-certs\") pod \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\" (UID: \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\") "
Oct 30 01:06:13.980459 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.980403    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-istio-podinfo\") pod \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\" (UID: \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\") "
Oct 30 01:06:13.980459 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.980435    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-istio-envoy\") pod \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\" (UID: \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\") "
Oct 30 01:06:13.980572 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.980473    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-istio-token\" (UniqueName: \"kubernetes.io/projected/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-istio-token\") pod \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\" (UID: \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\") "
Oct 30 01:06:13.980572 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.980511    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-kube-api-access-6cq5z\" (UniqueName: \"kubernetes.io/projected/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-kube-api-access-6cq5z\") pod \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\" (UID: \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\") "
Oct 30 01:06:13.980669 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.980572    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-credential-socket\") pod \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\" (UID: \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\") "
Oct 30 01:06:13.980669 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.980611    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-istiod-ca-cert\") pod \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\" (UID: \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\") "
Oct 30 01:06:13.980669 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.980648    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-istio-data\") pod \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\" (UID: \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\") "
Oct 30 01:06:13.980894 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.980877    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-workload-socket\") pod \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\" (UID: \"a8bdca6c-102d-4aaa-ab76-16a9123d6a74\") "
Oct 30 01:06:13.981406 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.981383    4393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-istio-podinfo\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:13.981532 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.981410    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5md4m\" (UniqueName: \"kubernetes.io/projected/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-kube-api-access-5md4m\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:13.981532 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.981424    4393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-istio-envoy\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:13.981532 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.981435    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:13.981532 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.981446    4393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604-workload-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:13.982254 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.982223    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-credential-socket" pod "a8bdca6c-102d-4aaa-ab76-16a9123d6a74" (UID: "a8bdca6c-102d-4aaa-ab76-16a9123d6a74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:13.982254 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.982231    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-istiod-ca-cert" pod "a8bdca6c-102d-4aaa-ab76-16a9123d6a74" (UID: "a8bdca6c-102d-4aaa-ab76-16a9123d6a74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:06:13.982636 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.982440    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-workload-certs" pod "a8bdca6c-102d-4aaa-ab76-16a9123d6a74" (UID: "a8bdca6c-102d-4aaa-ab76-16a9123d6a74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:13.982636 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.982573    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-istio-data" pod "a8bdca6c-102d-4aaa-ab76-16a9123d6a74" (UID: "a8bdca6c-102d-4aaa-ab76-16a9123d6a74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:13.984159 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.984124    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-workload-socket" pod "a8bdca6c-102d-4aaa-ab76-16a9123d6a74" (UID: "a8bdca6c-102d-4aaa-ab76-16a9123d6a74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:13.984849 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.984822    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-istio-podinfo" pod "a8bdca6c-102d-4aaa-ab76-16a9123d6a74" (UID: "a8bdca6c-102d-4aaa-ab76-16a9123d6a74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:06:13.985429 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.985402    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-kube-api-access-6cq5z" pod "a8bdca6c-102d-4aaa-ab76-16a9123d6a74" (UID: "a8bdca6c-102d-4aaa-ab76-16a9123d6a74"). InnerVolumeSpecName "kube-api-access-6cq5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:13.986449 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.986420    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-istio-token" pod "a8bdca6c-102d-4aaa-ab76-16a9123d6a74" (UID: "a8bdca6c-102d-4aaa-ab76-16a9123d6a74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:13.987904 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:13.987879    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-istio-envoy" pod "a8bdca6c-102d-4aaa-ab76-16a9123d6a74" (UID: "a8bdca6c-102d-4aaa-ab76-16a9123d6a74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:14.083844 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:14.082844    4393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-workload-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:14.083844 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:14.082884    4393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-workload-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:14.083844 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:14.082897    4393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-istio-podinfo\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:14.083844 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:14.082909    4393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-istio-envoy\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:14.083844 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:14.082921    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:14.083844 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:14.082933    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cq5z\" (UniqueName: \"kubernetes.io/projected/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-kube-api-access-6cq5z\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:14.083844 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:14.082948    4393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-credential-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:14.083844 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:14.082962    4393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-istiod-ca-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:14.083844 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:14.082974    4393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8bdca6c-102d-4aaa-ab76-16a9123d6a74-istio-data\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:14.451924 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:14.451884    4393 generic.go:356] "Generic (PLEG): container finished" podID="2bc5c9f7-dbe6-4741-8e37-258a1f1ed604" containerID="e0ca147f03a24f57bae14caf8fca4797d04d0c40c7e3fbd85fb71f3ea5d8fee0" exitCode=0
Oct 30 01:06:14.452404 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:14.451930    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"
Oct 30 01:06:14.452404 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:14.451967    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x" event={"ID":"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604","Type":"ContainerDied","Data":"e0ca147f03a24f57bae14caf8fca4797d04d0c40c7e3fbd85fb71f3ea5d8fee0"}
Oct 30 01:06:14.452404 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:14.452057    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x" event={"ID":"2bc5c9f7-dbe6-4741-8e37-258a1f1ed604","Type":"ContainerDied","Data":"978992d6f119023a29479f1c3012d8c69889361cd709c0f2ada6d53a6755443d"}
Oct 30 01:06:14.452404 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:14.452093    4393 scope.go:122] "RemoveContainer" containerID="e0ca147f03a24f57bae14caf8fca4797d04d0c40c7e3fbd85fb71f3ea5d8fee0"
Oct 30 01:06:14.454025 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:14.454001    4393 generic.go:356] "Generic (PLEG): container finished" podID="a8bdca6c-102d-4aaa-ab76-16a9123d6a74" containerID="434b71c4baabd19f38964a7311756922c8f8934b3b6c8beebe7df7898d3d01d8" exitCode=0
Oct 30 01:06:14.454306 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:14.454051    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg" event={"ID":"a8bdca6c-102d-4aaa-ab76-16a9123d6a74","Type":"ContainerDied","Data":"434b71c4baabd19f38964a7311756922c8f8934b3b6c8beebe7df7898d3d01d8"}
Oct 30 01:06:14.454306 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:14.454117    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg" event={"ID":"a8bdca6c-102d-4aaa-ab76-16a9123d6a74","Type":"ContainerDied","Data":"838a2a7c6018f42eaf1582f095167dd033fbf9e609ad5a884389fbb473eb7e2c"}
Oct 30 01:06:14.454306 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:14.454126    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"
Oct 30 01:06:14.464225 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:14.464205    4393 scope.go:122] "RemoveContainer" containerID="e0ca147f03a24f57bae14caf8fca4797d04d0c40c7e3fbd85fb71f3ea5d8fee0"
Oct 30 01:06:14.464488 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:06:14.464467    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0ca147f03a24f57bae14caf8fca4797d04d0c40c7e3fbd85fb71f3ea5d8fee0\": container with ID starting with e0ca147f03a24f57bae14caf8fca4797d04d0c40c7e3fbd85fb71f3ea5d8fee0 not found: ID does not exist" containerID="e0ca147f03a24f57bae14caf8fca4797d04d0c40c7e3fbd85fb71f3ea5d8fee0"
Oct 30 01:06:14.464541 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:14.464495    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0ca147f03a24f57bae14caf8fca4797d04d0c40c7e3fbd85fb71f3ea5d8fee0"} err="failed to get container status \"e0ca147f03a24f57bae14caf8fca4797d04d0c40c7e3fbd85fb71f3ea5d8fee0\": rpc error: code = NotFound desc = could not find container \"e0ca147f03a24f57bae14caf8fca4797d04d0c40c7e3fbd85fb71f3ea5d8fee0\": container with ID starting with e0ca147f03a24f57bae14caf8fca4797d04d0c40c7e3fbd85fb71f3ea5d8fee0 not found: ID does not exist"
Oct 30 01:06:14.464541 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:14.464513    4393 scope.go:122] "RemoveContainer" containerID="434b71c4baabd19f38964a7311756922c8f8934b3b6c8beebe7df7898d3d01d8"
Oct 30 01:06:14.474499 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:14.474481    4393 scope.go:122] "RemoveContainer" containerID="434b71c4baabd19f38964a7311756922c8f8934b3b6c8beebe7df7898d3d01d8"
Oct 30 01:06:14.474765 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:06:14.474747    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"434b71c4baabd19f38964a7311756922c8f8934b3b6c8beebe7df7898d3d01d8\": container with ID starting with 434b71c4baabd19f38964a7311756922c8f8934b3b6c8beebe7df7898d3d01d8 not found: ID does not exist" containerID="434b71c4baabd19f38964a7311756922c8f8934b3b6c8beebe7df7898d3d01d8"
Oct 30 01:06:14.474849 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:14.474772    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"434b71c4baabd19f38964a7311756922c8f8934b3b6c8beebe7df7898d3d01d8"} err="failed to get container status \"434b71c4baabd19f38964a7311756922c8f8934b3b6c8beebe7df7898d3d01d8\": rpc error: code = NotFound desc = could not find container \"434b71c4baabd19f38964a7311756922c8f8934b3b6c8beebe7df7898d3d01d8\": container with ID starting with 434b71c4baabd19f38964a7311756922c8f8934b3b6c8beebe7df7898d3d01d8 not found: ID does not exist"
Oct 30 01:06:14.517856 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:14.517834    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"]
Oct 30 01:06:14.529106 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:14.529086    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-4fz6x"]
Oct 30 01:06:14.559288 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:14.559255    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"]
Oct 30 01:06:14.563262 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:14.563238    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-vzqgg"]
Oct 30 01:06:15.728142 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:15.728106    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bc5c9f7-dbe6-4741-8e37-258a1f1ed604" path="/var/lib/kubelet/pods/2bc5c9f7-dbe6-4741-8e37-258a1f1ed604/volumes"
Oct 30 01:06:15.728688 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:15.728672    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8bdca6c-102d-4aaa-ab76-16a9123d6a74" path="/var/lib/kubelet/pods/a8bdca6c-102d-4aaa-ab76-16a9123d6a74/volumes"
Oct 30 01:06:16.380920 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:16.380883    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2"]
Oct 30 01:06:16.381157 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:16.381125    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2" podUID="eabc28b8-fab2-42e2-abc8-d69c3fd22959" containerName="maas-api" containerID="cri-o://511f10e569d7fd905c50200b0fd157704c0e663207529993b43f8356407ce895" gracePeriod=30
Oct 30 01:06:16.697467 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:16.697437    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2"
Oct 30 01:06:17.073856 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:17.072515    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eabc28b8-fab2-42e2-abc8-d69c3fd22959-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eabc28b8-fab2-42e2-abc8-d69c3fd22959-metrics-certs\") pod \"eabc28b8-fab2-42e2-abc8-d69c3fd22959\" (UID: \"eabc28b8-fab2-42e2-abc8-d69c3fd22959\") "
Oct 30 01:06:17.073856 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:17.072814    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eabc28b8-fab2-42e2-abc8-d69c3fd22959-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eabc28b8-fab2-42e2-abc8-d69c3fd22959-maas-api-tls\") pod \"eabc28b8-fab2-42e2-abc8-d69c3fd22959\" (UID: \"eabc28b8-fab2-42e2-abc8-d69c3fd22959\") "
Oct 30 01:06:17.073856 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:17.072950    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eabc28b8-fab2-42e2-abc8-d69c3fd22959-kube-api-access-lxlg8\" (UniqueName: \"kubernetes.io/projected/eabc28b8-fab2-42e2-abc8-d69c3fd22959-kube-api-access-lxlg8\") pod \"eabc28b8-fab2-42e2-abc8-d69c3fd22959\" (UID: \"eabc28b8-fab2-42e2-abc8-d69c3fd22959\") "
Oct 30 01:06:17.073856 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:17.073094    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eabc28b8-fab2-42e2-abc8-d69c3fd22959-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eabc28b8-fab2-42e2-abc8-d69c3fd22959-openshift-service-ca\") pod \"eabc28b8-fab2-42e2-abc8-d69c3fd22959\" (UID: \"eabc28b8-fab2-42e2-abc8-d69c3fd22959\") "
Oct 30 01:06:17.091823 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:17.087076    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eabc28b8-fab2-42e2-abc8-d69c3fd22959-openshift-service-ca" pod "eabc28b8-fab2-42e2-abc8-d69c3fd22959" (UID: "eabc28b8-fab2-42e2-abc8-d69c3fd22959"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:06:17.111993 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:17.109897    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eabc28b8-fab2-42e2-abc8-d69c3fd22959-kube-api-access-lxlg8" pod "eabc28b8-fab2-42e2-abc8-d69c3fd22959" (UID: "eabc28b8-fab2-42e2-abc8-d69c3fd22959"). InnerVolumeSpecName "kube-api-access-lxlg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:17.139130 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:17.139088    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eabc28b8-fab2-42e2-abc8-d69c3fd22959-metrics-certs" pod "eabc28b8-fab2-42e2-abc8-d69c3fd22959" (UID: "eabc28b8-fab2-42e2-abc8-d69c3fd22959"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:06:17.142358 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:17.142298    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eabc28b8-fab2-42e2-abc8-d69c3fd22959-maas-api-tls" pod "eabc28b8-fab2-42e2-abc8-d69c3fd22959" (UID: "eabc28b8-fab2-42e2-abc8-d69c3fd22959"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:06:17.175473 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:17.175436    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxlg8\" (UniqueName: \"kubernetes.io/projected/eabc28b8-fab2-42e2-abc8-d69c3fd22959-kube-api-access-lxlg8\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:17.175473 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:17.175468    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eabc28b8-fab2-42e2-abc8-d69c3fd22959-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:17.175473 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:17.175480    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eabc28b8-fab2-42e2-abc8-d69c3fd22959-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:17.175743 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:17.175489    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eabc28b8-fab2-42e2-abc8-d69c3fd22959-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:17.482097 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:17.482062    4393 generic.go:356] "Generic (PLEG): container finished" podID="eabc28b8-fab2-42e2-abc8-d69c3fd22959" containerID="511f10e569d7fd905c50200b0fd157704c0e663207529993b43f8356407ce895" exitCode=0
Oct 30 01:06:17.482271 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:17.482128    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2"
Oct 30 01:06:17.482271 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:17.482148    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2" event={"ID":"eabc28b8-fab2-42e2-abc8-d69c3fd22959","Type":"ContainerDied","Data":"511f10e569d7fd905c50200b0fd157704c0e663207529993b43f8356407ce895"}
Oct 30 01:06:17.482271 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:17.482203    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2" event={"ID":"eabc28b8-fab2-42e2-abc8-d69c3fd22959","Type":"ContainerDied","Data":"ff58159af840065eae9d3d233252fab6d54024ab7f7489e36f66cbe6c71bb8f3"}
Oct 30 01:06:17.482271 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:17.482228    4393 scope.go:122] "RemoveContainer" containerID="511f10e569d7fd905c50200b0fd157704c0e663207529993b43f8356407ce895"
Oct 30 01:06:17.494514 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:17.494497    4393 scope.go:122] "RemoveContainer" containerID="511f10e569d7fd905c50200b0fd157704c0e663207529993b43f8356407ce895"
Oct 30 01:06:17.494830 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:06:17.494806    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"511f10e569d7fd905c50200b0fd157704c0e663207529993b43f8356407ce895\": container with ID starting with 511f10e569d7fd905c50200b0fd157704c0e663207529993b43f8356407ce895 not found: ID does not exist" containerID="511f10e569d7fd905c50200b0fd157704c0e663207529993b43f8356407ce895"
Oct 30 01:06:17.494923 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:17.494842    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"511f10e569d7fd905c50200b0fd157704c0e663207529993b43f8356407ce895"} err="failed to get container status \"511f10e569d7fd905c50200b0fd157704c0e663207529993b43f8356407ce895\": rpc error: code = NotFound desc = could not find container \"511f10e569d7fd905c50200b0fd157704c0e663207529993b43f8356407ce895\": container with ID starting with 511f10e569d7fd905c50200b0fd157704c0e663207529993b43f8356407ce895 not found: ID does not exist"
Oct 30 01:06:17.535101 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:17.535074    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2"]
Oct 30 01:06:17.603173 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:17.603119    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5007e1-5996d87f44-b2gb2"]
Oct 30 01:06:17.727645 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:17.727564    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eabc28b8-fab2-42e2-abc8-d69c3fd22959" path="/var/lib/kubelet/pods/eabc28b8-fab2-42e2-abc8-d69c3fd22959/volumes"
Oct 30 01:06:22.820747 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:22.820714    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7f92f6-79df4f4c6c-c4qwp"]
Oct 30 01:06:22.821470 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:22.821454    4393 cpu_manager.go:414] "containerMap: removing container" podUID="a5b4591e-8713-40c5-b33c-e6505356ee29" containerName="istio-proxy"
Oct 30 01:06:22.821518 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:22.821471    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5b4591e-8713-40c5-b33c-e6505356ee29" containerName="istio-proxy"
Oct 30 01:06:22.821518 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:22.821498    4393 cpu_manager.go:414] "containerMap: removing container" podUID="2bc5c9f7-dbe6-4741-8e37-258a1f1ed604" containerName="istio-proxy"
Oct 30 01:06:22.821518 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:22.821503    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bc5c9f7-dbe6-4741-8e37-258a1f1ed604" containerName="istio-proxy"
Oct 30 01:06:22.821518 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:22.821514    4393 cpu_manager.go:414] "containerMap: removing container" podUID="eabc28b8-fab2-42e2-abc8-d69c3fd22959" containerName="maas-api"
Oct 30 01:06:22.821643 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:22.821519    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eabc28b8-fab2-42e2-abc8-d69c3fd22959" containerName="maas-api"
Oct 30 01:06:22.821643 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:22.821540    4393 cpu_manager.go:414] "containerMap: removing container" podUID="dc80bac8-5fa1-4c5e-bdb1-73225ad3181c" containerName="istio-proxy"
Oct 30 01:06:22.821643 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:22.821545    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc80bac8-5fa1-4c5e-bdb1-73225ad3181c" containerName="istio-proxy"
Oct 30 01:06:22.821643 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:22.821558    4393 cpu_manager.go:414] "containerMap: removing container" podUID="a8bdca6c-102d-4aaa-ab76-16a9123d6a74" containerName="istio-proxy"
Oct 30 01:06:22.821643 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:22.821563    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8bdca6c-102d-4aaa-ab76-16a9123d6a74" containerName="istio-proxy"
Oct 30 01:06:22.821811 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:22.821663    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc80bac8-5fa1-4c5e-bdb1-73225ad3181c" containerName="istio-proxy"
Oct 30 01:06:22.821811 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:22.821672    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8bdca6c-102d-4aaa-ab76-16a9123d6a74" containerName="istio-proxy"
Oct 30 01:06:22.821811 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:22.821680    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5b4591e-8713-40c5-b33c-e6505356ee29" containerName="istio-proxy"
Oct 30 01:06:22.821811 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:22.821690    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bc5c9f7-dbe6-4741-8e37-258a1f1ed604" containerName="istio-proxy"
Oct 30 01:06:22.821811 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:22.821698    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="eabc28b8-fab2-42e2-abc8-d69c3fd22959" containerName="maas-api"
Oct 30 01:06:22.827269 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:22.827243    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7f92f6-79df4f4c6c-c4qwp"
Oct 30 01:06:22.844028 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:22.843992    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7f92f6-79df4f4c6c-c4qwp"]
Oct 30 01:06:22.848683 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:22.848639    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-7f92f6\""
Oct 30 01:06:22.848683 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:22.848674    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-7f92f6-dockercfg-t569f\""
Oct 30 01:06:22.943804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:22.939307    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctm9l\" (UniqueName: \"kubernetes.io/projected/4debacaf-9ed1-4e85-a8f3-4f52a76ef705-kube-api-access-ctm9l\") pod \"payload-pre-processing-e2e-shared-b-w6-7f92f6-79df4f4c6c-c4qwp\" (UID: \"4debacaf-9ed1-4e85-a8f3-4f52a76ef705\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7f92f6-79df4f4c6c-c4qwp"
Oct 30 01:06:22.943804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:22.939751    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4debacaf-9ed1-4e85-a8f3-4f52a76ef705-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-7f92f6-79df4f4c6c-c4qwp\" (UID: \"4debacaf-9ed1-4e85-a8f3-4f52a76ef705\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7f92f6-79df4f4c6c-c4qwp"
Oct 30 01:06:22.990805 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:22.956770    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-7f92f6-5744474458-tvxhl"]
Oct 30 01:06:23.051562 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:23.051434    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4debacaf-9ed1-4e85-a8f3-4f52a76ef705-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-7f92f6-79df4f4c6c-c4qwp\" (UID: \"4debacaf-9ed1-4e85-a8f3-4f52a76ef705\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7f92f6-79df4f4c6c-c4qwp"
Oct 30 01:06:23.052095 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:23.052060    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctm9l\" (UniqueName: \"kubernetes.io/projected/4debacaf-9ed1-4e85-a8f3-4f52a76ef705-kube-api-access-ctm9l\") pod \"payload-pre-processing-e2e-shared-b-w6-7f92f6-79df4f4c6c-c4qwp\" (UID: \"4debacaf-9ed1-4e85-a8f3-4f52a76ef705\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7f92f6-79df4f4c6c-c4qwp"
Oct 30 01:06:23.057964 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:23.054163    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4debacaf-9ed1-4e85-a8f3-4f52a76ef705-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-7f92f6-79df4f4c6c-c4qwp\" (UID: \"4debacaf-9ed1-4e85-a8f3-4f52a76ef705\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7f92f6-79df4f4c6c-c4qwp"
Oct 30 01:06:23.072982 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:23.071626    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-7f92f6-5744474458-tvxhl"]
Oct 30 01:06:23.076632 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:23.071743    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-7f92f6-5744474458-tvxhl"
Oct 30 01:06:23.088854 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:23.085902    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctm9l\" (UniqueName: \"kubernetes.io/projected/4debacaf-9ed1-4e85-a8f3-4f52a76ef705-kube-api-access-ctm9l\") pod \"payload-pre-processing-e2e-shared-b-w6-7f92f6-79df4f4c6c-c4qwp\" (UID: \"4debacaf-9ed1-4e85-a8f3-4f52a76ef705\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7f92f6-79df4f4c6c-c4qwp"
Oct 30 01:06:23.144765 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:23.144724    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7f92f6-79df4f4c6c-c4qwp"
Oct 30 01:06:23.255662 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:23.255624    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5slr7\" (UniqueName: \"kubernetes.io/projected/a674fcd8-3f60-4f70-b199-a2ad09d74fb2-kube-api-access-5slr7\") pod \"payload-processing-e2e-shared-b-w6-7f92f6-5744474458-tvxhl\" (UID: \"a674fcd8-3f60-4f70-b199-a2ad09d74fb2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-7f92f6-5744474458-tvxhl"
Oct 30 01:06:23.256542 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:23.255839    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a674fcd8-3f60-4f70-b199-a2ad09d74fb2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-7f92f6-5744474458-tvxhl\" (UID: \"a674fcd8-3f60-4f70-b199-a2ad09d74fb2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-7f92f6-5744474458-tvxhl"
Oct 30 01:06:23.357318 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:23.357228    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a674fcd8-3f60-4f70-b199-a2ad09d74fb2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-7f92f6-5744474458-tvxhl\" (UID: \"a674fcd8-3f60-4f70-b199-a2ad09d74fb2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-7f92f6-5744474458-tvxhl"
Oct 30 01:06:23.357486 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:23.357405    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5slr7\" (UniqueName: \"kubernetes.io/projected/a674fcd8-3f60-4f70-b199-a2ad09d74fb2-kube-api-access-5slr7\") pod \"payload-processing-e2e-shared-b-w6-7f92f6-5744474458-tvxhl\" (UID: \"a674fcd8-3f60-4f70-b199-a2ad09d74fb2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-7f92f6-5744474458-tvxhl"
Oct 30 01:06:23.358888 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:23.358839    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a674fcd8-3f60-4f70-b199-a2ad09d74fb2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-7f92f6-5744474458-tvxhl\" (UID: \"a674fcd8-3f60-4f70-b199-a2ad09d74fb2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-7f92f6-5744474458-tvxhl"
Oct 30 01:06:23.380544 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:23.380515    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5slr7\" (UniqueName: \"kubernetes.io/projected/a674fcd8-3f60-4f70-b199-a2ad09d74fb2-kube-api-access-5slr7\") pod \"payload-processing-e2e-shared-b-w6-7f92f6-5744474458-tvxhl\" (UID: \"a674fcd8-3f60-4f70-b199-a2ad09d74fb2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-7f92f6-5744474458-tvxhl"
Oct 30 01:06:23.430974 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:23.430948    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-7f92f6-5744474458-tvxhl"
Oct 30 01:06:23.526109 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:23.523520    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7f92f6-79df4f4c6c-c4qwp"]
Oct 30 01:06:23.743064 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:06:23.743031    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda674fcd8_3f60_4f70_b199_a2ad09d74fb2.slice/crio-bb751936dcbfbe905c42186e264b539ba33387acbd8e8cd335850e1b521ea61e WatchSource:0}: Error finding container bb751936dcbfbe905c42186e264b539ba33387acbd8e8cd335850e1b521ea61e: Status 404 returned error can't find the container with id bb751936dcbfbe905c42186e264b539ba33387acbd8e8cd335850e1b521ea61e
Oct 30 01:06:23.743202 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:23.743132    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-7f92f6-5744474458-tvxhl"]
Oct 30 01:06:24.546859 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:24.546823    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-7f92f6-5744474458-tvxhl" event={"ID":"a674fcd8-3f60-4f70-b199-a2ad09d74fb2","Type":"ContainerStarted","Data":"fffac304f5df1d0f08fbbeb0ba526f0cfd4152caaee1dab2ec0906de2188b634"}
Oct 30 01:06:24.547331 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:24.546883    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-7f92f6-5744474458-tvxhl" event={"ID":"a674fcd8-3f60-4f70-b199-a2ad09d74fb2","Type":"ContainerStarted","Data":"bb751936dcbfbe905c42186e264b539ba33387acbd8e8cd335850e1b521ea61e"}
Oct 30 01:06:24.547331 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:24.546915    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-7f92f6-5744474458-tvxhl"
Oct 30 01:06:24.548709 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:24.548683    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7f92f6-79df4f4c6c-c4qwp" event={"ID":"4debacaf-9ed1-4e85-a8f3-4f52a76ef705","Type":"ContainerStarted","Data":"e81b1711ae3e8fccc1eab05ec101393d81a36a37e0873c67f5b288902a41e1b4"}
Oct 30 01:06:24.548880 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:24.548720    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7f92f6-79df4f4c6c-c4qwp" event={"ID":"4debacaf-9ed1-4e85-a8f3-4f52a76ef705","Type":"ContainerStarted","Data":"8fcacb0afbcf8b29dae3505e62d84b1771ce87a73a5c7f6dc4a8afb02aee824d"}
Oct 30 01:06:24.548880 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:24.548771    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7f92f6-79df4f4c6c-c4qwp"
Oct 30 01:06:24.587168 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:24.587119    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-7f92f6-5744474458-tvxhl" podStartSLOduration=2.587106007 podStartE2EDuration="2.587106007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:24.586421079 +0000 UTC m=+2074.974097469" watchObservedRunningTime="2026-10-30 01:06:24.587106007 +0000 UTC m=+2074.974782393"
Oct 30 01:06:24.628164 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:24.628120    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7f92f6-79df4f4c6c-c4qwp" podStartSLOduration=2.62808061 podStartE2EDuration="2.62808061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:24.625422596 +0000 UTC m=+2075.013098978" watchObservedRunningTime="2026-10-30 01:06:24.62808061 +0000 UTC m=+2075.015757050"
Oct 30 01:06:24.842636 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:24.842556    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj"]
Oct 30 01:06:24.848947 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:24.848870    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj"
Oct 30 01:06:24.868332 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:24.865440    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj"]
Oct 30 01:06:24.868332 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:24.867366    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-beeba4\"/\"openshift-service-ca.crt\""
Oct 30 01:06:24.868332 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:24.867506    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-beeba4\"/\"default-dockercfg-sgnxq\""
Oct 30 01:06:24.868332 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:24.867609    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-beeba4\"/\"auth-test-model-beeba4-kserve-self-signed-certs\""
Oct 30 01:06:24.868332 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:24.867694    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-beeba4\"/\"kube-root-ca.crt\""
Oct 30 01:06:24.985004 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:24.984971    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5c1b663-2684-4d99-bf83-75ff9c347e99-tmp-dir\") pod \"auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj\" (UID: \"e5c1b663-2684-4d99-bf83-75ff9c347e99\") " pod="ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj"
Oct 30 01:06:24.985181 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:24.985111    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5c1b663-2684-4d99-bf83-75ff9c347e99-home\") pod \"auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj\" (UID: \"e5c1b663-2684-4d99-bf83-75ff9c347e99\") " pod="ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj"
Oct 30 01:06:24.985181 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:24.985147    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5c1b663-2684-4d99-bf83-75ff9c347e99-tls-certs\") pod \"auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj\" (UID: \"e5c1b663-2684-4d99-bf83-75ff9c347e99\") " pod="ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj"
Oct 30 01:06:24.985308 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:24.985197    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5c1b663-2684-4d99-bf83-75ff9c347e99-dshm\") pod \"auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj\" (UID: \"e5c1b663-2684-4d99-bf83-75ff9c347e99\") " pod="ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj"
Oct 30 01:06:24.985355 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:24.985335    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5c1b663-2684-4d99-bf83-75ff9c347e99-model-cache\") pod \"auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj\" (UID: \"e5c1b663-2684-4d99-bf83-75ff9c347e99\") " pod="ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj"
Oct 30 01:06:24.985403 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:24.985391    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4npz\" (UniqueName: \"kubernetes.io/projected/e5c1b663-2684-4d99-bf83-75ff9c347e99-kube-api-access-g4npz\") pod \"auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj\" (UID: \"e5c1b663-2684-4d99-bf83-75ff9c347e99\") " pod="ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj"
Oct 30 01:06:25.086919 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:25.086863    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5c1b663-2684-4d99-bf83-75ff9c347e99-home\") pod \"auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj\" (UID: \"e5c1b663-2684-4d99-bf83-75ff9c347e99\") " pod="ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj"
Oct 30 01:06:25.087213 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:25.087189    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5c1b663-2684-4d99-bf83-75ff9c347e99-tls-certs\") pod \"auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj\" (UID: \"e5c1b663-2684-4d99-bf83-75ff9c347e99\") " pod="ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj"
Oct 30 01:06:25.087411 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:25.087380    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5c1b663-2684-4d99-bf83-75ff9c347e99-dshm\") pod \"auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj\" (UID: \"e5c1b663-2684-4d99-bf83-75ff9c347e99\") " pod="ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj"
Oct 30 01:06:25.087596 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:25.087579    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5c1b663-2684-4d99-bf83-75ff9c347e99-model-cache\") pod \"auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj\" (UID: \"e5c1b663-2684-4d99-bf83-75ff9c347e99\") " pod="ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj"
Oct 30 01:06:25.087755 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:25.087734    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4npz\" (UniqueName: \"kubernetes.io/projected/e5c1b663-2684-4d99-bf83-75ff9c347e99-kube-api-access-g4npz\") pod \"auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj\" (UID: \"e5c1b663-2684-4d99-bf83-75ff9c347e99\") " pod="ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj"
Oct 30 01:06:25.088007 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:25.087990    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5c1b663-2684-4d99-bf83-75ff9c347e99-tmp-dir\") pod \"auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj\" (UID: \"e5c1b663-2684-4d99-bf83-75ff9c347e99\") " pod="ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj"
Oct 30 01:06:25.088576 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:25.088556    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5c1b663-2684-4d99-bf83-75ff9c347e99-tmp-dir\") pod \"auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj\" (UID: \"e5c1b663-2684-4d99-bf83-75ff9c347e99\") " pod="ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj"
Oct 30 01:06:25.088964 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:25.088946    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5c1b663-2684-4d99-bf83-75ff9c347e99-home\") pod \"auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj\" (UID: \"e5c1b663-2684-4d99-bf83-75ff9c347e99\") " pod="ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj"
Oct 30 01:06:25.112844 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:25.106817    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5c1b663-2684-4d99-bf83-75ff9c347e99-dshm\") pod \"auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj\" (UID: \"e5c1b663-2684-4d99-bf83-75ff9c347e99\") " pod="ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj"
Oct 30 01:06:25.112844 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:25.107142    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5c1b663-2684-4d99-bf83-75ff9c347e99-model-cache\") pod \"auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj\" (UID: \"e5c1b663-2684-4d99-bf83-75ff9c347e99\") " pod="ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj"
Oct 30 01:06:25.117700 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:25.116488    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5c1b663-2684-4d99-bf83-75ff9c347e99-tls-certs\") pod \"auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj\" (UID: \"e5c1b663-2684-4d99-bf83-75ff9c347e99\") " pod="ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj"
Oct 30 01:06:25.149799 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:25.149533    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4npz\" (UniqueName: \"kubernetes.io/projected/e5c1b663-2684-4d99-bf83-75ff9c347e99-kube-api-access-g4npz\") pod \"auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj\" (UID: \"e5c1b663-2684-4d99-bf83-75ff9c347e99\") " pod="ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj"
Oct 30 01:06:25.184735 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:25.184432    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj"
Oct 30 01:06:25.560026 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:06:25.559997    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5c1b663_2684_4d99_bf83_75ff9c347e99.slice/crio-7619d4971ac06d51739c064c128093f91da1a2865b56a777acb345df517ab13a WatchSource:0}: Error finding container 7619d4971ac06d51739c064c128093f91da1a2865b56a777acb345df517ab13a: Status 404 returned error can't find the container with id 7619d4971ac06d51739c064c128093f91da1a2865b56a777acb345df517ab13a
Oct 30 01:06:25.560406 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:25.560135    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj"]
Oct 30 01:06:26.513322 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:06:26.513286    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:06:26.568895 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:26.568863    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj" event={"ID":"e5c1b663-2684-4d99-bf83-75ff9c347e99","Type":"ContainerStarted","Data":"aaf16a755305154e3a5b0e88aa6040f66bd7d027d244ca98342906d227f922f1"}
Oct 30 01:06:26.569283 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:26.568902    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj" event={"ID":"e5c1b663-2684-4d99-bf83-75ff9c347e99","Type":"ContainerStarted","Data":"7619d4971ac06d51739c064c128093f91da1a2865b56a777acb345df517ab13a"}
Oct 30 01:06:26.610961 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:26.610904    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj" podStartSLOduration=2.610888417 podStartE2EDuration="2.610888417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:26.608708329 +0000 UTC m=+2076.996384719" watchObservedRunningTime="2026-10-30 01:06:26.610888417 +0000 UTC m=+2076.998564860"
Oct 30 01:06:27.188461 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:27.188425    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj"
Oct 30 01:06:34.884020 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:34.883986    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn"]
Oct 30 01:06:34.889742 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:34.889717    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn"
Oct 30 01:06:34.914817 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:34.911712    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn"]
Oct 30 01:06:34.915880 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:34.915736    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-5f387c\"/\"kube-root-ca.crt\""
Oct 30 01:06:34.915880 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:34.915819    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-5f387c\"/\"openshift-service-ca.crt\""
Oct 30 01:06:34.915880 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:34.915840    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-5f387c\"/\"default-dockercfg-vr87d\""
Oct 30 01:06:34.915880 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:34.915881    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-5f387c\"/\"auth-test-model-5f387c-kserve-self-signed-certs\""
Oct 30 01:06:34.962292 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:34.962255    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73d9db41-7564-4d70-b061-bcb981ca5263-tmp-dir\") pod \"auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn\" (UID: \"73d9db41-7564-4d70-b061-bcb981ca5263\") " pod="ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn"
Oct 30 01:06:34.962447 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:34.962328    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73d9db41-7564-4d70-b061-bcb981ca5263-home\") pod \"auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn\" (UID: \"73d9db41-7564-4d70-b061-bcb981ca5263\") " pod="ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn"
Oct 30 01:06:34.962487 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:34.962448    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73d9db41-7564-4d70-b061-bcb981ca5263-tls-certs\") pod \"auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn\" (UID: \"73d9db41-7564-4d70-b061-bcb981ca5263\") " pod="ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn"
Oct 30 01:06:34.962526 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:34.962489    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73d9db41-7564-4d70-b061-bcb981ca5263-model-cache\") pod \"auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn\" (UID: \"73d9db41-7564-4d70-b061-bcb981ca5263\") " pod="ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn"
Oct 30 01:06:34.962612 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:34.962599    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73d9db41-7564-4d70-b061-bcb981ca5263-dshm\") pod \"auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn\" (UID: \"73d9db41-7564-4d70-b061-bcb981ca5263\") " pod="ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn"
Oct 30 01:06:34.962656 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:34.962645    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mpbz\" (UniqueName: \"kubernetes.io/projected/73d9db41-7564-4d70-b061-bcb981ca5263-kube-api-access-9mpbz\") pod \"auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn\" (UID: \"73d9db41-7564-4d70-b061-bcb981ca5263\") " pod="ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn"
Oct 30 01:06:35.063835 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:35.063798    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73d9db41-7564-4d70-b061-bcb981ca5263-dshm\") pod \"auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn\" (UID: \"73d9db41-7564-4d70-b061-bcb981ca5263\") " pod="ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn"
Oct 30 01:06:35.064013 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:35.063868    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mpbz\" (UniqueName: \"kubernetes.io/projected/73d9db41-7564-4d70-b061-bcb981ca5263-kube-api-access-9mpbz\") pod \"auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn\" (UID: \"73d9db41-7564-4d70-b061-bcb981ca5263\") " pod="ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn"
Oct 30 01:06:35.064013 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:35.063933    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73d9db41-7564-4d70-b061-bcb981ca5263-tmp-dir\") pod \"auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn\" (UID: \"73d9db41-7564-4d70-b061-bcb981ca5263\") " pod="ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn"
Oct 30 01:06:35.064133 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:35.064019    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73d9db41-7564-4d70-b061-bcb981ca5263-home\") pod \"auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn\" (UID: \"73d9db41-7564-4d70-b061-bcb981ca5263\") " pod="ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn"
Oct 30 01:06:35.064133 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:35.064114    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73d9db41-7564-4d70-b061-bcb981ca5263-tls-certs\") pod \"auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn\" (UID: \"73d9db41-7564-4d70-b061-bcb981ca5263\") " pod="ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn"
Oct 30 01:06:35.064237 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:35.064160    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73d9db41-7564-4d70-b061-bcb981ca5263-model-cache\") pod \"auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn\" (UID: \"73d9db41-7564-4d70-b061-bcb981ca5263\") " pod="ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn"
Oct 30 01:06:35.064423 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:35.064395    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73d9db41-7564-4d70-b061-bcb981ca5263-tmp-dir\") pod \"auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn\" (UID: \"73d9db41-7564-4d70-b061-bcb981ca5263\") " pod="ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn"
Oct 30 01:06:35.064741 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:35.064434    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73d9db41-7564-4d70-b061-bcb981ca5263-home\") pod \"auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn\" (UID: \"73d9db41-7564-4d70-b061-bcb981ca5263\") " pod="ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn"
Oct 30 01:06:35.064741 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:35.064512    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73d9db41-7564-4d70-b061-bcb981ca5263-model-cache\") pod \"auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn\" (UID: \"73d9db41-7564-4d70-b061-bcb981ca5263\") " pod="ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn"
Oct 30 01:06:35.066355 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:35.066335    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73d9db41-7564-4d70-b061-bcb981ca5263-dshm\") pod \"auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn\" (UID: \"73d9db41-7564-4d70-b061-bcb981ca5263\") " pod="ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn"
Oct 30 01:06:35.066802 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:35.066763    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73d9db41-7564-4d70-b061-bcb981ca5263-tls-certs\") pod \"auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn\" (UID: \"73d9db41-7564-4d70-b061-bcb981ca5263\") " pod="ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn"
Oct 30 01:06:35.086676 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:35.086650    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mpbz\" (UniqueName: \"kubernetes.io/projected/73d9db41-7564-4d70-b061-bcb981ca5263-kube-api-access-9mpbz\") pod \"auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn\" (UID: \"73d9db41-7564-4d70-b061-bcb981ca5263\") " pod="ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn"
Oct 30 01:06:35.190479 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:35.189204    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj"
Oct 30 01:06:35.221839 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:35.221064    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn"
Oct 30 01:06:35.265423 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:35.265317    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj"
Oct 30 01:06:35.566618 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:35.566518    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-7f92f6-5744474458-tvxhl"
Oct 30 01:06:35.567939 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:35.567919    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7f92f6-79df4f4c6c-c4qwp"
Oct 30 01:06:35.614176 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:35.613117    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn"]
Oct 30 01:06:35.674824 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:35.673311    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn" event={"ID":"73d9db41-7564-4d70-b061-bcb981ca5263","Type":"ContainerStarted","Data":"1c97cb727910650982dad691e2c7b23d3d41074bb25089d8bdaff0f6268f4d87"}
Oct 30 01:06:35.694390 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:35.694361    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj"
Oct 30 01:06:36.690458 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:36.690409    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn" event={"ID":"73d9db41-7564-4d70-b061-bcb981ca5263","Type":"ContainerStarted","Data":"b981c58d0524971abd9d2bb5fe2ff599452cdbc456bc4521bf25edb7ea60930c"}
Oct 30 01:06:36.732439 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:36.732385    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn" podStartSLOduration=2.732370912 podStartE2EDuration="2.732370912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:36.729904781 +0000 UTC m=+2087.117581170" watchObservedRunningTime="2026-10-30 01:06:36.732370912 +0000 UTC m=+2087.120047331"
Oct 30 01:06:37.221696 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:37.221655    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn"
Oct 30 01:06:45.222015 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:45.221980    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn"
Oct 30 01:06:45.234669 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:45.234640    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn"
Oct 30 01:06:45.812573 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:45.812540    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn"
Oct 30 01:06:48.749067 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:48.749011    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"]
Oct 30 01:06:48.749746 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:48.749262    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d" podUID="5f7dc119-7915-473a-b425-b9fdd217a8e8" containerName="istio-proxy" containerID="cri-o://9f43ac339255b8065d3f28e2906c10d443fe70595a7a00ae80e0b501d6797bf5" gracePeriod=30
Oct 30 01:06:53.385392 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:53.385355    4393 patch_prober.go:29] interesting pod/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:06:53.385917 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:53.385418    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d" podUID="5f7dc119-7915-473a-b425-b9fdd217a8e8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:06:54.052253 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.052225    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"
Oct 30 01:06:54.160856 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.154718    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f7dc119-7915-473a-b425-b9fdd217a8e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f7dc119-7915-473a-b425-b9fdd217a8e8-istio-data\") pod \"5f7dc119-7915-473a-b425-b9fdd217a8e8\" (UID: \"5f7dc119-7915-473a-b425-b9fdd217a8e8\") "
Oct 30 01:06:54.160856 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.154821    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f7dc119-7915-473a-b425-b9fdd217a8e8-istio-token\" (UniqueName: \"kubernetes.io/projected/5f7dc119-7915-473a-b425-b9fdd217a8e8-istio-token\") pod \"5f7dc119-7915-473a-b425-b9fdd217a8e8\" (UID: \"5f7dc119-7915-473a-b425-b9fdd217a8e8\") "
Oct 30 01:06:54.160856 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.154850    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f7dc119-7915-473a-b425-b9fdd217a8e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7dc119-7915-473a-b425-b9fdd217a8e8-workload-socket\") pod \"5f7dc119-7915-473a-b425-b9fdd217a8e8\" (UID: \"5f7dc119-7915-473a-b425-b9fdd217a8e8\") "
Oct 30 01:06:54.160856 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.154930    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f7dc119-7915-473a-b425-b9fdd217a8e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f7dc119-7915-473a-b425-b9fdd217a8e8-istio-podinfo\") pod \"5f7dc119-7915-473a-b425-b9fdd217a8e8\" (UID: \"5f7dc119-7915-473a-b425-b9fdd217a8e8\") "
Oct 30 01:06:54.160856 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.155010    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f7dc119-7915-473a-b425-b9fdd217a8e8-kube-api-access-fzfph\" (UniqueName: \"kubernetes.io/projected/5f7dc119-7915-473a-b425-b9fdd217a8e8-kube-api-access-fzfph\") pod \"5f7dc119-7915-473a-b425-b9fdd217a8e8\" (UID: \"5f7dc119-7915-473a-b425-b9fdd217a8e8\") "
Oct 30 01:06:54.160856 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.155135    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f7dc119-7915-473a-b425-b9fdd217a8e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f7dc119-7915-473a-b425-b9fdd217a8e8-istio-envoy\") pod \"5f7dc119-7915-473a-b425-b9fdd217a8e8\" (UID: \"5f7dc119-7915-473a-b425-b9fdd217a8e8\") "
Oct 30 01:06:54.160856 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.155230    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f7dc119-7915-473a-b425-b9fdd217a8e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f7dc119-7915-473a-b425-b9fdd217a8e8-workload-certs\") pod \"5f7dc119-7915-473a-b425-b9fdd217a8e8\" (UID: \"5f7dc119-7915-473a-b425-b9fdd217a8e8\") "
Oct 30 01:06:54.160856 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.155428    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f7dc119-7915-473a-b425-b9fdd217a8e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f7dc119-7915-473a-b425-b9fdd217a8e8-istiod-ca-cert\") pod \"5f7dc119-7915-473a-b425-b9fdd217a8e8\" (UID: \"5f7dc119-7915-473a-b425-b9fdd217a8e8\") "
Oct 30 01:06:54.160856 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.155466    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f7dc119-7915-473a-b425-b9fdd217a8e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7dc119-7915-473a-b425-b9fdd217a8e8-credential-socket\") pod \"5f7dc119-7915-473a-b425-b9fdd217a8e8\" (UID: \"5f7dc119-7915-473a-b425-b9fdd217a8e8\") "
Oct 30 01:06:54.167167 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.165391    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f7dc119-7915-473a-b425-b9fdd217a8e8-credential-socket" pod "5f7dc119-7915-473a-b425-b9fdd217a8e8" (UID: "5f7dc119-7915-473a-b425-b9fdd217a8e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:54.167447 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.167415    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f7dc119-7915-473a-b425-b9fdd217a8e8-istio-data" pod "5f7dc119-7915-473a-b425-b9fdd217a8e8" (UID: "5f7dc119-7915-473a-b425-b9fdd217a8e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:54.168303 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.168274    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f7dc119-7915-473a-b425-b9fdd217a8e8-workload-certs" pod "5f7dc119-7915-473a-b425-b9fdd217a8e8" (UID: "5f7dc119-7915-473a-b425-b9fdd217a8e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:54.169297 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.169256    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f7dc119-7915-473a-b425-b9fdd217a8e8-workload-socket" pod "5f7dc119-7915-473a-b425-b9fdd217a8e8" (UID: "5f7dc119-7915-473a-b425-b9fdd217a8e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:54.175034 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.175010    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f7dc119-7915-473a-b425-b9fdd217a8e8-istiod-ca-cert" pod "5f7dc119-7915-473a-b425-b9fdd217a8e8" (UID: "5f7dc119-7915-473a-b425-b9fdd217a8e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:06:54.186077 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.185929    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f7dc119-7915-473a-b425-b9fdd217a8e8-kube-api-access-fzfph" pod "5f7dc119-7915-473a-b425-b9fdd217a8e8" (UID: "5f7dc119-7915-473a-b425-b9fdd217a8e8"). InnerVolumeSpecName "kube-api-access-fzfph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:54.186077 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.186049    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f7dc119-7915-473a-b425-b9fdd217a8e8-istio-token" pod "5f7dc119-7915-473a-b425-b9fdd217a8e8" (UID: "5f7dc119-7915-473a-b425-b9fdd217a8e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:06:54.186426 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.186399    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f7dc119-7915-473a-b425-b9fdd217a8e8-istio-envoy" pod "5f7dc119-7915-473a-b425-b9fdd217a8e8" (UID: "5f7dc119-7915-473a-b425-b9fdd217a8e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:06:54.188359 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.188330    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f7dc119-7915-473a-b425-b9fdd217a8e8-istio-podinfo" pod "5f7dc119-7915-473a-b425-b9fdd217a8e8" (UID: "5f7dc119-7915-473a-b425-b9fdd217a8e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:06:54.257173 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.257141    4393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7dc119-7915-473a-b425-b9fdd217a8e8-credential-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:54.257173 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.257171    4393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f7dc119-7915-473a-b425-b9fdd217a8e8-istio-data\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:54.257173 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.257181    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f7dc119-7915-473a-b425-b9fdd217a8e8-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:54.257397 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.257189    4393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f7dc119-7915-473a-b425-b9fdd217a8e8-workload-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:54.257397 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.257197    4393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f7dc119-7915-473a-b425-b9fdd217a8e8-istio-podinfo\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:54.257397 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.257207    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzfph\" (UniqueName: \"kubernetes.io/projected/5f7dc119-7915-473a-b425-b9fdd217a8e8-kube-api-access-fzfph\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:54.257397 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.257215    4393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f7dc119-7915-473a-b425-b9fdd217a8e8-istio-envoy\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:54.257397 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.257222    4393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f7dc119-7915-473a-b425-b9fdd217a8e8-workload-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:54.257397 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.257230    4393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f7dc119-7915-473a-b425-b9fdd217a8e8-istiod-ca-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:06:54.875484 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.875447    4393 generic.go:356] "Generic (PLEG): container finished" podID="5f7dc119-7915-473a-b425-b9fdd217a8e8" containerID="9f43ac339255b8065d3f28e2906c10d443fe70595a7a00ae80e0b501d6797bf5" exitCode=0
Oct 30 01:06:54.876021 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.875508    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"
Oct 30 01:06:54.876021 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.875531    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d" event={"ID":"5f7dc119-7915-473a-b425-b9fdd217a8e8","Type":"ContainerDied","Data":"9f43ac339255b8065d3f28e2906c10d443fe70595a7a00ae80e0b501d6797bf5"}
Oct 30 01:06:54.876021 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.875592    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d" event={"ID":"5f7dc119-7915-473a-b425-b9fdd217a8e8","Type":"ContainerDied","Data":"cbc9fdff77178b0b0d4cdd61ea6b614e013233a531e6c2b1d59693bb92ae7a03"}
Oct 30 01:06:54.876021 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.875649    4393 scope.go:122] "RemoveContainer" containerID="9f43ac339255b8065d3f28e2906c10d443fe70595a7a00ae80e0b501d6797bf5"
Oct 30 01:06:54.888477 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.888458    4393 scope.go:122] "RemoveContainer" containerID="9f43ac339255b8065d3f28e2906c10d443fe70595a7a00ae80e0b501d6797bf5"
Oct 30 01:06:54.888824 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:06:54.888799    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f43ac339255b8065d3f28e2906c10d443fe70595a7a00ae80e0b501d6797bf5\": container with ID starting with 9f43ac339255b8065d3f28e2906c10d443fe70595a7a00ae80e0b501d6797bf5 not found: ID does not exist" containerID="9f43ac339255b8065d3f28e2906c10d443fe70595a7a00ae80e0b501d6797bf5"
Oct 30 01:06:54.888896 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.888833    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f43ac339255b8065d3f28e2906c10d443fe70595a7a00ae80e0b501d6797bf5"} err="failed to get container status \"9f43ac339255b8065d3f28e2906c10d443fe70595a7a00ae80e0b501d6797bf5\": rpc error: code = NotFound desc = could not find container \"9f43ac339255b8065d3f28e2906c10d443fe70595a7a00ae80e0b501d6797bf5\": container with ID starting with 9f43ac339255b8065d3f28e2906c10d443fe70595a7a00ae80e0b501d6797bf5 not found: ID does not exist"
Oct 30 01:06:54.940626 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.940598    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"]
Oct 30 01:06:54.945196 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:54.945169    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b1f6ed-openshift-default-767786fdd5-ghc4d"]
Oct 30 01:06:55.727735 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:55.727702    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f7dc119-7915-473a-b425-b9fdd217a8e8" path="/var/lib/kubelet/pods/5f7dc119-7915-473a-b425-b9fdd217a8e8/volumes"
Oct 30 01:06:55.916748 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:55.916712    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm"]
Oct 30 01:06:55.917547 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:55.917529    4393 cpu_manager.go:414] "containerMap: removing container" podUID="5f7dc119-7915-473a-b425-b9fdd217a8e8" containerName="istio-proxy"
Oct 30 01:06:55.917593 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:55.917550    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f7dc119-7915-473a-b425-b9fdd217a8e8" containerName="istio-proxy"
Oct 30 01:06:55.917708 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:55.917698    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f7dc119-7915-473a-b425-b9fdd217a8e8" containerName="istio-proxy"
Oct 30 01:06:55.922961 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:55.922940    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm"
Oct 30 01:06:55.952406 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:55.952379    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-7f92f6-serving-cert\""
Oct 30 01:06:55.959660 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:55.959631    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm"]
Oct 30 01:06:56.083665 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:56.083576    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6da14397-0216-4ada-83bf-72addba282e5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm\" (UID: \"6da14397-0216-4ada-83bf-72addba282e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm"
Oct 30 01:06:56.083665 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:56.083627    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6da14397-0216-4ada-83bf-72addba282e5-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm\" (UID: \"6da14397-0216-4ada-83bf-72addba282e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm"
Oct 30 01:06:56.083935 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:56.083834    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px8k4\" (UniqueName: \"kubernetes.io/projected/6da14397-0216-4ada-83bf-72addba282e5-kube-api-access-px8k4\") pod \"maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm\" (UID: \"6da14397-0216-4ada-83bf-72addba282e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm"
Oct 30 01:06:56.083993 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:56.083959    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6da14397-0216-4ada-83bf-72addba282e5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm\" (UID: \"6da14397-0216-4ada-83bf-72addba282e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm"
Oct 30 01:06:56.185324 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:56.185285    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6da14397-0216-4ada-83bf-72addba282e5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm\" (UID: \"6da14397-0216-4ada-83bf-72addba282e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm"
Oct 30 01:06:56.185510 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:56.185409    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6da14397-0216-4ada-83bf-72addba282e5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm\" (UID: \"6da14397-0216-4ada-83bf-72addba282e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm"
Oct 30 01:06:56.185510 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:56.185445    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6da14397-0216-4ada-83bf-72addba282e5-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm\" (UID: \"6da14397-0216-4ada-83bf-72addba282e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm"
Oct 30 01:06:56.185640 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:56.185533    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px8k4\" (UniqueName: \"kubernetes.io/projected/6da14397-0216-4ada-83bf-72addba282e5-kube-api-access-px8k4\") pod \"maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm\" (UID: \"6da14397-0216-4ada-83bf-72addba282e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm"
Oct 30 01:06:56.186015 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:56.185986    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6da14397-0216-4ada-83bf-72addba282e5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm\" (UID: \"6da14397-0216-4ada-83bf-72addba282e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm"
Oct 30 01:06:56.188386 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:56.188363    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6da14397-0216-4ada-83bf-72addba282e5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm\" (UID: \"6da14397-0216-4ada-83bf-72addba282e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm"
Oct 30 01:06:56.188505 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:56.188392    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6da14397-0216-4ada-83bf-72addba282e5-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm\" (UID: \"6da14397-0216-4ada-83bf-72addba282e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm"
Oct 30 01:06:56.208681 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:56.208656    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px8k4\" (UniqueName: \"kubernetes.io/projected/6da14397-0216-4ada-83bf-72addba282e5-kube-api-access-px8k4\") pod \"maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm\" (UID: \"6da14397-0216-4ada-83bf-72addba282e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm"
Oct 30 01:06:56.238965 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:56.238915    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm"
Oct 30 01:06:56.523395 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:06:56.523356    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:06:56.733757 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:06:56.733728    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6da14397_0216_4ada_83bf_72addba282e5.slice/crio-db4d4f671e46ab0a063eafacd79847b42a0e8535681b9d8ee95eb38a01422fde WatchSource:0}: Error finding container db4d4f671e46ab0a063eafacd79847b42a0e8535681b9d8ee95eb38a01422fde: Status 404 returned error can't find the container with id db4d4f671e46ab0a063eafacd79847b42a0e8535681b9d8ee95eb38a01422fde
Oct 30 01:06:56.734488 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:56.734459    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm"]
Oct 30 01:06:56.895208 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:56.895175    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm" event={"ID":"6da14397-0216-4ada-83bf-72addba282e5","Type":"ContainerStarted","Data":"db4d4f671e46ab0a063eafacd79847b42a0e8535681b9d8ee95eb38a01422fde"}
Oct 30 01:06:57.906238 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:57.906199    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm" event={"ID":"6da14397-0216-4ada-83bf-72addba282e5","Type":"ContainerStarted","Data":"372a4304cb68e907993b4246e0387ce0ef6354e4e02fce81581686e46e325d45"}
Oct 30 01:06:57.906717 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:57.906271    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm"
Oct 30 01:06:57.948400 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:06:57.948346    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm" podStartSLOduration=2.565367294 podStartE2EDuration="2.948330415s" totalImagesPullingTime="382.963121ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:06:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:06:57.946117058 +0000 UTC m=+2108.333793447" watchObservedRunningTime="2026-10-30 01:06:57.948330415 +0000 UTC m=+2108.336006803"
Oct 30 01:07:00.939583 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:00.939490    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn"]
Oct 30 01:07:00.945739 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:00.945714    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn"
Oct 30 01:07:00.965317 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:00.965289    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn"]
Oct 30 01:07:00.968535 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:00.968510    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8e0f66\"/\"kube-root-ca.crt\""
Oct 30 01:07:00.972330 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:00.972308    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8e0f66\"/\"openshift-service-ca.crt\""
Oct 30 01:07:00.972466 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:00.972431    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8e0f66\"/\"default-dockercfg-2z847\""
Oct 30 01:07:00.972568 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:00.972540    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8e0f66\"/\"sub-test-model-8e0f66-kserve-self-signed-certs\""
Oct 30 01:07:01.133710 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:01.133671    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-tmp-dir\") pod \"sub-test-model-8e0f66-kserve-74dffc749f-nhfqn\" (UID: \"ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2\") " pod="ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn"
Oct 30 01:07:01.133710 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:01.133712    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-home\") pod \"sub-test-model-8e0f66-kserve-74dffc749f-nhfqn\" (UID: \"ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2\") " pod="ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn"
Oct 30 01:07:01.133984 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:01.133900    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-dshm\") pod \"sub-test-model-8e0f66-kserve-74dffc749f-nhfqn\" (UID: \"ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2\") " pod="ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn"
Oct 30 01:07:01.133984 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:01.133929    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-model-cache\") pod \"sub-test-model-8e0f66-kserve-74dffc749f-nhfqn\" (UID: \"ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2\") " pod="ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn"
Oct 30 01:07:01.133984 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:01.133961    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-tls-certs\") pod \"sub-test-model-8e0f66-kserve-74dffc749f-nhfqn\" (UID: \"ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2\") " pod="ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn"
Oct 30 01:07:01.134205 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:01.134185    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8wfd\" (UniqueName: \"kubernetes.io/projected/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-kube-api-access-r8wfd\") pod \"sub-test-model-8e0f66-kserve-74dffc749f-nhfqn\" (UID: \"ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2\") " pod="ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn"
Oct 30 01:07:01.237859 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:01.236799    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-dshm\") pod \"sub-test-model-8e0f66-kserve-74dffc749f-nhfqn\" (UID: \"ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2\") " pod="ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn"
Oct 30 01:07:01.237859 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:01.236857    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-model-cache\") pod \"sub-test-model-8e0f66-kserve-74dffc749f-nhfqn\" (UID: \"ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2\") " pod="ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn"
Oct 30 01:07:01.237859 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:01.236906    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-tls-certs\") pod \"sub-test-model-8e0f66-kserve-74dffc749f-nhfqn\" (UID: \"ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2\") " pod="ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn"
Oct 30 01:07:01.237859 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:01.237090    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8wfd\" (UniqueName: \"kubernetes.io/projected/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-kube-api-access-r8wfd\") pod \"sub-test-model-8e0f66-kserve-74dffc749f-nhfqn\" (UID: \"ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2\") " pod="ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn"
Oct 30 01:07:01.237859 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:01.237223    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-tmp-dir\") pod \"sub-test-model-8e0f66-kserve-74dffc749f-nhfqn\" (UID: \"ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2\") " pod="ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn"
Oct 30 01:07:01.237859 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:01.237258    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-home\") pod \"sub-test-model-8e0f66-kserve-74dffc749f-nhfqn\" (UID: \"ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2\") " pod="ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn"
Oct 30 01:07:01.238345 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:01.237957    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-home\") pod \"sub-test-model-8e0f66-kserve-74dffc749f-nhfqn\" (UID: \"ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2\") " pod="ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn"
Oct 30 01:07:01.247850 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:01.246209    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-model-cache\") pod \"sub-test-model-8e0f66-kserve-74dffc749f-nhfqn\" (UID: \"ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2\") " pod="ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn"
Oct 30 01:07:01.247850 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:01.246859    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-tmp-dir\") pod \"sub-test-model-8e0f66-kserve-74dffc749f-nhfqn\" (UID: \"ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2\") " pod="ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn"
Oct 30 01:07:01.270961 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:01.269157    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-tls-certs\") pod \"sub-test-model-8e0f66-kserve-74dffc749f-nhfqn\" (UID: \"ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2\") " pod="ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn"
Oct 30 01:07:01.273172 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:01.273141    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-dshm\") pod \"sub-test-model-8e0f66-kserve-74dffc749f-nhfqn\" (UID: \"ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2\") " pod="ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn"
Oct 30 01:07:01.275508 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:01.275469    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8wfd\" (UniqueName: \"kubernetes.io/projected/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-kube-api-access-r8wfd\") pod \"sub-test-model-8e0f66-kserve-74dffc749f-nhfqn\" (UID: \"ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2\") " pod="ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn"
Oct 30 01:07:01.567638 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:01.567563    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn"
Oct 30 01:07:01.873624 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:07:01.873597    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podddc80cf1_533f_4b1c_8d51_cdef7ec8f1c2.slice/crio-8a516e765a04bf936a0defab518856b5608f98361d6ea7e91361b313bc7cf7b9 WatchSource:0}: Error finding container 8a516e765a04bf936a0defab518856b5608f98361d6ea7e91361b313bc7cf7b9: Status 404 returned error can't find the container with id 8a516e765a04bf936a0defab518856b5608f98361d6ea7e91361b313bc7cf7b9
Oct 30 01:07:01.874898 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:01.874868    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn"]
Oct 30 01:07:01.955449 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:01.955417    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn" event={"ID":"ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2","Type":"ContainerStarted","Data":"85a40f9a35d83c9ba1ac70f65eda02937a982365859998f9514f7c3bf02730be"}
Oct 30 01:07:01.955887 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:01.955465    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn" event={"ID":"ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2","Type":"ContainerStarted","Data":"8a516e765a04bf936a0defab518856b5608f98361d6ea7e91361b313bc7cf7b9"}
Oct 30 01:07:01.996180 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:01.996112    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn" podStartSLOduration=1.996096959 podStartE2EDuration="1.996096959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:01.994052904 +0000 UTC m=+2112.381729292" watchObservedRunningTime="2026-10-30 01:07:01.996096959 +0000 UTC m=+2112.383773347"
Oct 30 01:07:02.568556 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:02.568524    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn"
Oct 30 01:07:03.920758 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:03.920727    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm"
Oct 30 01:07:11.568001 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:11.567966    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn"
Oct 30 01:07:11.581013 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:11.580986    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn"
Oct 30 01:07:11.824674 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:11.824588    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz"]
Oct 30 01:07:11.830798 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:11.830763    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz"
Oct 30 01:07:11.846167 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:11.846136    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-73435a\"/\"kube-root-ca.crt\""
Oct 30 01:07:11.847688 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:11.846928    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-73435a\"/\"default-dockercfg-w9cql\""
Oct 30 01:07:11.847688 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:11.847035    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-73435a\"/\"auth-test-model-73435a-kserve-self-signed-certs\""
Oct 30 01:07:11.847688 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:11.847115    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-73435a\"/\"openshift-service-ca.crt\""
Oct 30 01:07:11.849530 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:11.849507    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz"]
Oct 30 01:07:11.903892 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:11.903860    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc530394-4e4b-4d7f-b031-65139f579756-tls-certs\") pod \"auth-test-model-73435a-kserve-6cf8b665-2frgz\" (UID: \"bc530394-4e4b-4d7f-b031-65139f579756\") " pod="ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz"
Oct 30 01:07:11.904055 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:11.903902    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc530394-4e4b-4d7f-b031-65139f579756-home\") pod \"auth-test-model-73435a-kserve-6cf8b665-2frgz\" (UID: \"bc530394-4e4b-4d7f-b031-65139f579756\") " pod="ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz"
Oct 30 01:07:11.904055 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:11.903986    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc530394-4e4b-4d7f-b031-65139f579756-model-cache\") pod \"auth-test-model-73435a-kserve-6cf8b665-2frgz\" (UID: \"bc530394-4e4b-4d7f-b031-65139f579756\") " pod="ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz"
Oct 30 01:07:11.904055 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:11.904043    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9chhr\" (UniqueName: \"kubernetes.io/projected/bc530394-4e4b-4d7f-b031-65139f579756-kube-api-access-9chhr\") pod \"auth-test-model-73435a-kserve-6cf8b665-2frgz\" (UID: \"bc530394-4e4b-4d7f-b031-65139f579756\") " pod="ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz"
Oct 30 01:07:11.904166 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:11.904134    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc530394-4e4b-4d7f-b031-65139f579756-dshm\") pod \"auth-test-model-73435a-kserve-6cf8b665-2frgz\" (UID: \"bc530394-4e4b-4d7f-b031-65139f579756\") " pod="ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz"
Oct 30 01:07:11.904200 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:11.904182    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc530394-4e4b-4d7f-b031-65139f579756-tmp-dir\") pod \"auth-test-model-73435a-kserve-6cf8b665-2frgz\" (UID: \"bc530394-4e4b-4d7f-b031-65139f579756\") " pod="ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz"
Oct 30 01:07:12.005421 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:12.005385    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc530394-4e4b-4d7f-b031-65139f579756-dshm\") pod \"auth-test-model-73435a-kserve-6cf8b665-2frgz\" (UID: \"bc530394-4e4b-4d7f-b031-65139f579756\") " pod="ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz"
Oct 30 01:07:12.005635 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:12.005440    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc530394-4e4b-4d7f-b031-65139f579756-tmp-dir\") pod \"auth-test-model-73435a-kserve-6cf8b665-2frgz\" (UID: \"bc530394-4e4b-4d7f-b031-65139f579756\") " pod="ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz"
Oct 30 01:07:12.005635 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:12.005602    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc530394-4e4b-4d7f-b031-65139f579756-tls-certs\") pod \"auth-test-model-73435a-kserve-6cf8b665-2frgz\" (UID: \"bc530394-4e4b-4d7f-b031-65139f579756\") " pod="ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz"
Oct 30 01:07:12.005758 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:12.005638    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc530394-4e4b-4d7f-b031-65139f579756-home\") pod \"auth-test-model-73435a-kserve-6cf8b665-2frgz\" (UID: \"bc530394-4e4b-4d7f-b031-65139f579756\") " pod="ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz"
Oct 30 01:07:12.005758 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:12.005695    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc530394-4e4b-4d7f-b031-65139f579756-model-cache\") pod \"auth-test-model-73435a-kserve-6cf8b665-2frgz\" (UID: \"bc530394-4e4b-4d7f-b031-65139f579756\") " pod="ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz"
Oct 30 01:07:12.005758 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:12.005725    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9chhr\" (UniqueName: \"kubernetes.io/projected/bc530394-4e4b-4d7f-b031-65139f579756-kube-api-access-9chhr\") pod \"auth-test-model-73435a-kserve-6cf8b665-2frgz\" (UID: \"bc530394-4e4b-4d7f-b031-65139f579756\") " pod="ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz"
Oct 30 01:07:12.006001 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:12.005979    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc530394-4e4b-4d7f-b031-65139f579756-tmp-dir\") pod \"auth-test-model-73435a-kserve-6cf8b665-2frgz\" (UID: \"bc530394-4e4b-4d7f-b031-65139f579756\") " pod="ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz"
Oct 30 01:07:12.006082 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:12.006057    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc530394-4e4b-4d7f-b031-65139f579756-home\") pod \"auth-test-model-73435a-kserve-6cf8b665-2frgz\" (UID: \"bc530394-4e4b-4d7f-b031-65139f579756\") " pod="ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz"
Oct 30 01:07:12.006139 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:12.006109    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc530394-4e4b-4d7f-b031-65139f579756-model-cache\") pod \"auth-test-model-73435a-kserve-6cf8b665-2frgz\" (UID: \"bc530394-4e4b-4d7f-b031-65139f579756\") " pod="ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz"
Oct 30 01:07:12.007948 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:12.007926    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc530394-4e4b-4d7f-b031-65139f579756-dshm\") pod \"auth-test-model-73435a-kserve-6cf8b665-2frgz\" (UID: \"bc530394-4e4b-4d7f-b031-65139f579756\") " pod="ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz"
Oct 30 01:07:12.008275 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:12.008259    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc530394-4e4b-4d7f-b031-65139f579756-tls-certs\") pod \"auth-test-model-73435a-kserve-6cf8b665-2frgz\" (UID: \"bc530394-4e4b-4d7f-b031-65139f579756\") " pod="ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz"
Oct 30 01:07:12.028741 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:12.028710    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9chhr\" (UniqueName: \"kubernetes.io/projected/bc530394-4e4b-4d7f-b031-65139f579756-kube-api-access-9chhr\") pod \"auth-test-model-73435a-kserve-6cf8b665-2frgz\" (UID: \"bc530394-4e4b-4d7f-b031-65139f579756\") " pod="ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz"
Oct 30 01:07:12.057734 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:12.057703    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn"
Oct 30 01:07:12.161146 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:12.161109    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz"
Oct 30 01:07:12.521598 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:07:12.521539    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc530394_4e4b_4d7f_b031_65139f579756.slice/crio-11ed37d329ce730fdf8692bb4a9e20a4f1d855d12bf2d53dfce85d68ff450e6e WatchSource:0}: Error finding container 11ed37d329ce730fdf8692bb4a9e20a4f1d855d12bf2d53dfce85d68ff450e6e: Status 404 returned error can't find the container with id 11ed37d329ce730fdf8692bb4a9e20a4f1d855d12bf2d53dfce85d68ff450e6e
Oct 30 01:07:12.521748 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:12.521711    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz"]
Oct 30 01:07:13.053322 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:13.053277    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz" event={"ID":"bc530394-4e4b-4d7f-b031-65139f579756","Type":"ContainerStarted","Data":"1885f5b74627a6ef2f702a1f1c0ba3a37598f6b4d37859c4e9b8cfb2d98bc520"}
Oct 30 01:07:13.053695 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:13.053332    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz" event={"ID":"bc530394-4e4b-4d7f-b031-65139f579756","Type":"ContainerStarted","Data":"11ed37d329ce730fdf8692bb4a9e20a4f1d855d12bf2d53dfce85d68ff450e6e"}
Oct 30 01:07:13.095678 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:13.095627    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz" podStartSLOduration=2.095612754 podStartE2EDuration="2.095612754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:13.092882358 +0000 UTC m=+2123.480558746" watchObservedRunningTime="2026-10-30 01:07:13.095612754 +0000 UTC m=+2123.483289190"
Oct 30 01:07:13.161932 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:13.161900    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz"
Oct 30 01:07:13.604171 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:13.604118    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g"]
Oct 30 01:07:13.610287 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:13.610260    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g"
Oct 30 01:07:13.635379 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:13.635305    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-00e07a\"/\"kube-root-ca.crt\""
Oct 30 01:07:13.641627 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:13.641600    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-00e07a\"/\"default-dockercfg-gtz2v\""
Oct 30 01:07:13.641819 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:13.641751    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-00e07a\"/\"auth-test-model-00e07a-kserve-self-signed-certs\""
Oct 30 01:07:13.648691 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:13.648658    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g"]
Oct 30 01:07:13.739288 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:13.739260    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-00e07a\"/\"openshift-service-ca.crt\""
Oct 30 01:07:13.741225 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:13.741195    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/931977c3-8e00-4f75-b85b-be0666416b82-tls-certs\") pod \"auth-test-model-00e07a-kserve-594589575c-kd75g\" (UID: \"931977c3-8e00-4f75-b85b-be0666416b82\") " pod="ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g"
Oct 30 01:07:13.741358 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:13.741246    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkwdg\" (UniqueName: \"kubernetes.io/projected/931977c3-8e00-4f75-b85b-be0666416b82-kube-api-access-tkwdg\") pod \"auth-test-model-00e07a-kserve-594589575c-kd75g\" (UID: \"931977c3-8e00-4f75-b85b-be0666416b82\") " pod="ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g"
Oct 30 01:07:13.741406 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:13.741368    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/931977c3-8e00-4f75-b85b-be0666416b82-home\") pod \"auth-test-model-00e07a-kserve-594589575c-kd75g\" (UID: \"931977c3-8e00-4f75-b85b-be0666416b82\") " pod="ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g"
Oct 30 01:07:13.741518 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:13.741500    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/931977c3-8e00-4f75-b85b-be0666416b82-dshm\") pod \"auth-test-model-00e07a-kserve-594589575c-kd75g\" (UID: \"931977c3-8e00-4f75-b85b-be0666416b82\") " pod="ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g"
Oct 30 01:07:13.741605 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:13.741586    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/931977c3-8e00-4f75-b85b-be0666416b82-tmp-dir\") pod \"auth-test-model-00e07a-kserve-594589575c-kd75g\" (UID: \"931977c3-8e00-4f75-b85b-be0666416b82\") " pod="ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g"
Oct 30 01:07:13.741661 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:13.741622    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/931977c3-8e00-4f75-b85b-be0666416b82-model-cache\") pod \"auth-test-model-00e07a-kserve-594589575c-kd75g\" (UID: \"931977c3-8e00-4f75-b85b-be0666416b82\") " pod="ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g"
Oct 30 01:07:13.843051 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:13.843014    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/931977c3-8e00-4f75-b85b-be0666416b82-home\") pod \"auth-test-model-00e07a-kserve-594589575c-kd75g\" (UID: \"931977c3-8e00-4f75-b85b-be0666416b82\") " pod="ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g"
Oct 30 01:07:13.843216 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:13.843143    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/931977c3-8e00-4f75-b85b-be0666416b82-dshm\") pod \"auth-test-model-00e07a-kserve-594589575c-kd75g\" (UID: \"931977c3-8e00-4f75-b85b-be0666416b82\") " pod="ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g"
Oct 30 01:07:13.843216 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:13.843187    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/931977c3-8e00-4f75-b85b-be0666416b82-tmp-dir\") pod \"auth-test-model-00e07a-kserve-594589575c-kd75g\" (UID: \"931977c3-8e00-4f75-b85b-be0666416b82\") " pod="ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g"
Oct 30 01:07:13.843319 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:13.843213    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/931977c3-8e00-4f75-b85b-be0666416b82-model-cache\") pod \"auth-test-model-00e07a-kserve-594589575c-kd75g\" (UID: \"931977c3-8e00-4f75-b85b-be0666416b82\") " pod="ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g"
Oct 30 01:07:13.843916 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:13.843522    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/931977c3-8e00-4f75-b85b-be0666416b82-tls-certs\") pod \"auth-test-model-00e07a-kserve-594589575c-kd75g\" (UID: \"931977c3-8e00-4f75-b85b-be0666416b82\") " pod="ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g"
Oct 30 01:07:13.843916 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:13.843569    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/931977c3-8e00-4f75-b85b-be0666416b82-home\") pod \"auth-test-model-00e07a-kserve-594589575c-kd75g\" (UID: \"931977c3-8e00-4f75-b85b-be0666416b82\") " pod="ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g"
Oct 30 01:07:13.843916 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:13.843603    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkwdg\" (UniqueName: \"kubernetes.io/projected/931977c3-8e00-4f75-b85b-be0666416b82-kube-api-access-tkwdg\") pod \"auth-test-model-00e07a-kserve-594589575c-kd75g\" (UID: \"931977c3-8e00-4f75-b85b-be0666416b82\") " pod="ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g"
Oct 30 01:07:13.843916 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:13.843615    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/931977c3-8e00-4f75-b85b-be0666416b82-model-cache\") pod \"auth-test-model-00e07a-kserve-594589575c-kd75g\" (UID: \"931977c3-8e00-4f75-b85b-be0666416b82\") " pod="ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g"
Oct 30 01:07:13.843916 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:13.843653    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/931977c3-8e00-4f75-b85b-be0666416b82-tmp-dir\") pod \"auth-test-model-00e07a-kserve-594589575c-kd75g\" (UID: \"931977c3-8e00-4f75-b85b-be0666416b82\") " pod="ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g"
Oct 30 01:07:13.845617 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:13.845596    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/931977c3-8e00-4f75-b85b-be0666416b82-dshm\") pod \"auth-test-model-00e07a-kserve-594589575c-kd75g\" (UID: \"931977c3-8e00-4f75-b85b-be0666416b82\") " pod="ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g"
Oct 30 01:07:13.846098 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:13.846080    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/931977c3-8e00-4f75-b85b-be0666416b82-tls-certs\") pod \"auth-test-model-00e07a-kserve-594589575c-kd75g\" (UID: \"931977c3-8e00-4f75-b85b-be0666416b82\") " pod="ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g"
Oct 30 01:07:13.867113 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:13.867044    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkwdg\" (UniqueName: \"kubernetes.io/projected/931977c3-8e00-4f75-b85b-be0666416b82-kube-api-access-tkwdg\") pod \"auth-test-model-00e07a-kserve-594589575c-kd75g\" (UID: \"931977c3-8e00-4f75-b85b-be0666416b82\") " pod="ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g"
Oct 30 01:07:13.945804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:13.939110    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g"
Oct 30 01:07:14.325089 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:07:14.325045    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod931977c3_8e00_4f75_b85b_be0666416b82.slice/crio-bf7ef8fa20842ecee478db91c21d0b2da88d7221d790977a7c7dc77e319a13ea WatchSource:0}: Error finding container bf7ef8fa20842ecee478db91c21d0b2da88d7221d790977a7c7dc77e319a13ea: Status 404 returned error can't find the container with id bf7ef8fa20842ecee478db91c21d0b2da88d7221d790977a7c7dc77e319a13ea
Oct 30 01:07:14.325963 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:14.325944    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g"]
Oct 30 01:07:15.077844 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:15.077810    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g" event={"ID":"931977c3-8e00-4f75-b85b-be0666416b82","Type":"ContainerStarted","Data":"7ad63c330103adc0cb599c86656b30d66de6e49810027addd507a05a372fadaa"}
Oct 30 01:07:15.078018 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:15.077854    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g" event={"ID":"931977c3-8e00-4f75-b85b-be0666416b82","Type":"ContainerStarted","Data":"bf7ef8fa20842ecee478db91c21d0b2da88d7221d790977a7c7dc77e319a13ea"}
Oct 30 01:07:15.120578 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:15.120528    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g" podStartSLOduration=2.120514508 podStartE2EDuration="2.120514508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:15.117225597 +0000 UTC m=+2125.504901989" watchObservedRunningTime="2026-10-30 01:07:15.120514508 +0000 UTC m=+2125.508190896"
Oct 30 01:07:15.940429 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:15.940396    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g"
Oct 30 01:07:22.161954 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:22.161920    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz"
Oct 30 01:07:22.174736 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:22.174700    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz"
Oct 30 01:07:23.189546 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:23.189517    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz"
Oct 30 01:07:23.940142 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:23.940108    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g"
Oct 30 01:07:23.952730 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:23.952699    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g"
Oct 30 01:07:24.189925 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:24.189890    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g"
Oct 30 01:07:26.512406 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:07:26.512371    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:07:31.832367 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:31.832335    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"]
Oct 30 01:07:31.836737 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:31.836715    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"
Oct 30 01:07:31.866442 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:31.866407    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"]
Oct 30 01:07:31.942986 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:31.942952    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-7f92f6-openshift-default-dockercfg-n4bqm\""
Oct 30 01:07:32.023584 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:32.023555    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4czjt\" (UniqueName: \"kubernetes.io/projected/07623c58-4919-43fe-ad87-6e47c6af9596-kube-api-access-4czjt\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4\" (UID: \"07623c58-4919-43fe-ad87-6e47c6af9596\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"
Oct 30 01:07:32.023741 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:32.023602    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07623c58-4919-43fe-ad87-6e47c6af9596-credential-socket\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4\" (UID: \"07623c58-4919-43fe-ad87-6e47c6af9596\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"
Oct 30 01:07:32.023823 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:32.023754    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07623c58-4919-43fe-ad87-6e47c6af9596-istio-data\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4\" (UID: \"07623c58-4919-43fe-ad87-6e47c6af9596\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"
Oct 30 01:07:32.023890 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:32.023873    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07623c58-4919-43fe-ad87-6e47c6af9596-workload-certs\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4\" (UID: \"07623c58-4919-43fe-ad87-6e47c6af9596\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"
Oct 30 01:07:32.023950 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:32.023935    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07623c58-4919-43fe-ad87-6e47c6af9596-istio-envoy\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4\" (UID: \"07623c58-4919-43fe-ad87-6e47c6af9596\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"
Oct 30 01:07:32.024023 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:32.024008    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07623c58-4919-43fe-ad87-6e47c6af9596-istiod-ca-cert\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4\" (UID: \"07623c58-4919-43fe-ad87-6e47c6af9596\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"
Oct 30 01:07:32.024155 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:32.024131    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07623c58-4919-43fe-ad87-6e47c6af9596-istio-token\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4\" (UID: \"07623c58-4919-43fe-ad87-6e47c6af9596\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"
Oct 30 01:07:32.024234 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:32.024217    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07623c58-4919-43fe-ad87-6e47c6af9596-istio-podinfo\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4\" (UID: \"07623c58-4919-43fe-ad87-6e47c6af9596\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"
Oct 30 01:07:32.024336 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:32.024320    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07623c58-4919-43fe-ad87-6e47c6af9596-workload-socket\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4\" (UID: \"07623c58-4919-43fe-ad87-6e47c6af9596\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"
Oct 30 01:07:32.125571 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:32.125540    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07623c58-4919-43fe-ad87-6e47c6af9596-istio-data\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4\" (UID: \"07623c58-4919-43fe-ad87-6e47c6af9596\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"
Oct 30 01:07:32.125761 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:32.125595    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07623c58-4919-43fe-ad87-6e47c6af9596-workload-certs\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4\" (UID: \"07623c58-4919-43fe-ad87-6e47c6af9596\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"
Oct 30 01:07:32.125761 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:32.125642    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07623c58-4919-43fe-ad87-6e47c6af9596-istio-envoy\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4\" (UID: \"07623c58-4919-43fe-ad87-6e47c6af9596\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"
Oct 30 01:07:32.125761 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:32.125695    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07623c58-4919-43fe-ad87-6e47c6af9596-istiod-ca-cert\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4\" (UID: \"07623c58-4919-43fe-ad87-6e47c6af9596\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"
Oct 30 01:07:32.125761 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:32.125741    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07623c58-4919-43fe-ad87-6e47c6af9596-istio-token\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4\" (UID: \"07623c58-4919-43fe-ad87-6e47c6af9596\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"
Oct 30 01:07:32.126055 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:32.125824    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07623c58-4919-43fe-ad87-6e47c6af9596-istio-podinfo\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4\" (UID: \"07623c58-4919-43fe-ad87-6e47c6af9596\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"
Oct 30 01:07:32.126055 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:32.125910    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07623c58-4919-43fe-ad87-6e47c6af9596-workload-socket\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4\" (UID: \"07623c58-4919-43fe-ad87-6e47c6af9596\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"
Oct 30 01:07:32.126055 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:32.125967    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4czjt\" (UniqueName: \"kubernetes.io/projected/07623c58-4919-43fe-ad87-6e47c6af9596-kube-api-access-4czjt\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4\" (UID: \"07623c58-4919-43fe-ad87-6e47c6af9596\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"
Oct 30 01:07:32.126055 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:32.126011    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07623c58-4919-43fe-ad87-6e47c6af9596-credential-socket\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4\" (UID: \"07623c58-4919-43fe-ad87-6e47c6af9596\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"
Oct 30 01:07:32.126055 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:32.126040    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07623c58-4919-43fe-ad87-6e47c6af9596-workload-certs\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4\" (UID: \"07623c58-4919-43fe-ad87-6e47c6af9596\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"
Oct 30 01:07:32.126305 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:32.126244    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07623c58-4919-43fe-ad87-6e47c6af9596-workload-socket\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4\" (UID: \"07623c58-4919-43fe-ad87-6e47c6af9596\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"
Oct 30 01:07:32.126365 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:32.126007    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07623c58-4919-43fe-ad87-6e47c6af9596-istio-data\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4\" (UID: \"07623c58-4919-43fe-ad87-6e47c6af9596\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"
Oct 30 01:07:32.126556 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:32.126526    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07623c58-4919-43fe-ad87-6e47c6af9596-credential-socket\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4\" (UID: \"07623c58-4919-43fe-ad87-6e47c6af9596\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"
Oct 30 01:07:32.126739 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:32.126715    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07623c58-4919-43fe-ad87-6e47c6af9596-istiod-ca-cert\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4\" (UID: \"07623c58-4919-43fe-ad87-6e47c6af9596\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"
Oct 30 01:07:32.128726 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:32.128703    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07623c58-4919-43fe-ad87-6e47c6af9596-istio-envoy\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4\" (UID: \"07623c58-4919-43fe-ad87-6e47c6af9596\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"
Oct 30 01:07:32.128985 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:32.128963    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07623c58-4919-43fe-ad87-6e47c6af9596-istio-podinfo\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4\" (UID: \"07623c58-4919-43fe-ad87-6e47c6af9596\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"
Oct 30 01:07:32.152241 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:32.152205    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07623c58-4919-43fe-ad87-6e47c6af9596-istio-token\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4\" (UID: \"07623c58-4919-43fe-ad87-6e47c6af9596\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"
Oct 30 01:07:32.152558 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:32.152539    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4czjt\" (UniqueName: \"kubernetes.io/projected/07623c58-4919-43fe-ad87-6e47c6af9596-kube-api-access-4czjt\") pod \"e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4\" (UID: \"07623c58-4919-43fe-ad87-6e47c6af9596\") " pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"
Oct 30 01:07:32.154483 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:32.154453    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"
Oct 30 01:07:32.490153 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:32.490126    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"]
Oct 30 01:07:32.490476 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:07:32.490455    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07623c58_4919_43fe_ad87_6e47c6af9596.slice/crio-21edc06989304625a97153a41baa07e3e5433d097e982abfb56da102fc8e29a2 WatchSource:0}: Error finding container 21edc06989304625a97153a41baa07e3e5433d097e982abfb56da102fc8e29a2: Status 404 returned error can't find the container with id 21edc06989304625a97153a41baa07e3e5433d097e982abfb56da102fc8e29a2
Oct 30 01:07:32.493648 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:32.493607    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:07:32.493759 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:32.493695    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:07:32.493759 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:32.493738    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:07:33.290116 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:33.290073    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4" event={"ID":"07623c58-4919-43fe-ad87-6e47c6af9596","Type":"ContainerStarted","Data":"11c43132f9f9288dd204660a7f2b236d8dfd20585d9301d92c58b2533f39a6b2"}
Oct 30 01:07:33.290647 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:33.290132    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4" event={"ID":"07623c58-4919-43fe-ad87-6e47c6af9596","Type":"ContainerStarted","Data":"21edc06989304625a97153a41baa07e3e5433d097e982abfb56da102fc8e29a2"}
Oct 30 01:07:33.363861 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:33.363808    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4" podStartSLOduration=2.363792037 podStartE2EDuration="2.363792037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:07:33.359674151 +0000 UTC m=+2143.747350540" watchObservedRunningTime="2026-10-30 01:07:33.363792037 +0000 UTC m=+2143.751468424"
Oct 30 01:07:34.154962 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:34.154926    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"
Oct 30 01:07:34.156941 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:34.156914    4393 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 01:07:34.157076 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:34.156962    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4" podUID="07623c58-4919-43fe-ad87-6e47c6af9596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 01:07:35.155866 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:35.155832    4393 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 01:07:35.156268 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:35.155886    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4" podUID="07623c58-4919-43fe-ad87-6e47c6af9596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 01:07:36.155480 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:36.155446    4393 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 01:07:36.155682 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:36.155513    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4" podUID="07623c58-4919-43fe-ad87-6e47c6af9596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 01:07:37.155612 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:37.155573    4393 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 01:07:37.156131 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:37.155633    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4" podUID="07623c58-4919-43fe-ad87-6e47c6af9596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 01:07:38.155695 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:38.155663    4393 patch_prober.go:29] interesting pod/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 01:07:38.156159 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:38.155722    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4" podUID="07623c58-4919-43fe-ad87-6e47c6af9596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 01:07:39.160255 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:39.160215    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"
Oct 30 01:07:39.160680 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:39.160661    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"
Oct 30 01:07:39.161477 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:07:39.161460    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"
Oct 30 01:07:56.513058 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:07:56.513014    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:08:14.972972 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:14.972936    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn"]
Oct 30 01:08:20.291761 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:20.291720    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"]
Oct 30 01:08:20.400845 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:20.398621    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"]
Oct 30 01:08:20.400845 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:20.398762    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"
Oct 30 01:08:20.414048 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:20.413994    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-93079c-openshift-default-dockercfg-9g4x2\""
Oct 30 01:08:20.452349 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:20.452295    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5355e22-6596-4c63-a492-fe8b578f04b5-istiod-ca-cert\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72\" (UID: \"d5355e22-6596-4c63-a492-fe8b578f04b5\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"
Oct 30 01:08:20.452806 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:20.452755    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5355e22-6596-4c63-a492-fe8b578f04b5-credential-socket\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72\" (UID: \"d5355e22-6596-4c63-a492-fe8b578f04b5\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"
Oct 30 01:08:20.452929 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:20.452897    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh26r\" (UniqueName: \"kubernetes.io/projected/d5355e22-6596-4c63-a492-fe8b578f04b5-kube-api-access-zh26r\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72\" (UID: \"d5355e22-6596-4c63-a492-fe8b578f04b5\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"
Oct 30 01:08:20.453016 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:20.452995    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5355e22-6596-4c63-a492-fe8b578f04b5-istio-data\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72\" (UID: \"d5355e22-6596-4c63-a492-fe8b578f04b5\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"
Oct 30 01:08:20.454914 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:20.453037    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5355e22-6596-4c63-a492-fe8b578f04b5-istio-podinfo\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72\" (UID: \"d5355e22-6596-4c63-a492-fe8b578f04b5\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"
Oct 30 01:08:20.454914 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:20.453270    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5355e22-6596-4c63-a492-fe8b578f04b5-workload-socket\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72\" (UID: \"d5355e22-6596-4c63-a492-fe8b578f04b5\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"
Oct 30 01:08:20.454914 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:20.453335    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5355e22-6596-4c63-a492-fe8b578f04b5-workload-certs\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72\" (UID: \"d5355e22-6596-4c63-a492-fe8b578f04b5\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"
Oct 30 01:08:20.454914 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:20.453419    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5355e22-6596-4c63-a492-fe8b578f04b5-istio-envoy\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72\" (UID: \"d5355e22-6596-4c63-a492-fe8b578f04b5\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"
Oct 30 01:08:20.454914 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:20.453459    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5355e22-6596-4c63-a492-fe8b578f04b5-istio-token\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72\" (UID: \"d5355e22-6596-4c63-a492-fe8b578f04b5\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"
Oct 30 01:08:20.555610 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:20.555527    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5355e22-6596-4c63-a492-fe8b578f04b5-istiod-ca-cert\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72\" (UID: \"d5355e22-6596-4c63-a492-fe8b578f04b5\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"
Oct 30 01:08:20.555816 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:20.555614    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5355e22-6596-4c63-a492-fe8b578f04b5-credential-socket\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72\" (UID: \"d5355e22-6596-4c63-a492-fe8b578f04b5\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"
Oct 30 01:08:20.555816 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:20.555712    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh26r\" (UniqueName: \"kubernetes.io/projected/d5355e22-6596-4c63-a492-fe8b578f04b5-kube-api-access-zh26r\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72\" (UID: \"d5355e22-6596-4c63-a492-fe8b578f04b5\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"
Oct 30 01:08:20.555816 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:20.555807    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5355e22-6596-4c63-a492-fe8b578f04b5-istio-data\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72\" (UID: \"d5355e22-6596-4c63-a492-fe8b578f04b5\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"
Oct 30 01:08:20.555999 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:20.555835    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5355e22-6596-4c63-a492-fe8b578f04b5-istio-podinfo\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72\" (UID: \"d5355e22-6596-4c63-a492-fe8b578f04b5\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"
Oct 30 01:08:20.555999 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:20.555952    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5355e22-6596-4c63-a492-fe8b578f04b5-workload-socket\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72\" (UID: \"d5355e22-6596-4c63-a492-fe8b578f04b5\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"
Oct 30 01:08:20.556113 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:20.556008    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5355e22-6596-4c63-a492-fe8b578f04b5-workload-certs\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72\" (UID: \"d5355e22-6596-4c63-a492-fe8b578f04b5\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"
Oct 30 01:08:20.556113 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:20.556077    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5355e22-6596-4c63-a492-fe8b578f04b5-istio-envoy\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72\" (UID: \"d5355e22-6596-4c63-a492-fe8b578f04b5\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"
Oct 30 01:08:20.556228 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:20.556115    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5355e22-6596-4c63-a492-fe8b578f04b5-istio-token\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72\" (UID: \"d5355e22-6596-4c63-a492-fe8b578f04b5\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"
Oct 30 01:08:20.556228 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:20.556139    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5355e22-6596-4c63-a492-fe8b578f04b5-credential-socket\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72\" (UID: \"d5355e22-6596-4c63-a492-fe8b578f04b5\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"
Oct 30 01:08:20.556339 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:20.556241    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5355e22-6596-4c63-a492-fe8b578f04b5-istio-data\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72\" (UID: \"d5355e22-6596-4c63-a492-fe8b578f04b5\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"
Oct 30 01:08:20.556391 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:20.556366    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5355e22-6596-4c63-a492-fe8b578f04b5-workload-socket\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72\" (UID: \"d5355e22-6596-4c63-a492-fe8b578f04b5\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"
Oct 30 01:08:20.556443 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:20.556414    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5355e22-6596-4c63-a492-fe8b578f04b5-workload-certs\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72\" (UID: \"d5355e22-6596-4c63-a492-fe8b578f04b5\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"
Oct 30 01:08:20.556515 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:20.556441    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5355e22-6596-4c63-a492-fe8b578f04b5-istiod-ca-cert\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72\" (UID: \"d5355e22-6596-4c63-a492-fe8b578f04b5\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"
Oct 30 01:08:20.558703 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:20.558676    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5355e22-6596-4c63-a492-fe8b578f04b5-istio-envoy\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72\" (UID: \"d5355e22-6596-4c63-a492-fe8b578f04b5\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"
Oct 30 01:08:20.558837 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:20.558736    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5355e22-6596-4c63-a492-fe8b578f04b5-istio-podinfo\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72\" (UID: \"d5355e22-6596-4c63-a492-fe8b578f04b5\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"
Oct 30 01:08:20.581076 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:20.581037    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh26r\" (UniqueName: \"kubernetes.io/projected/d5355e22-6596-4c63-a492-fe8b578f04b5-kube-api-access-zh26r\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72\" (UID: \"d5355e22-6596-4c63-a492-fe8b578f04b5\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"
Oct 30 01:08:20.581412 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:20.581370    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5355e22-6596-4c63-a492-fe8b578f04b5-istio-token\") pod \"e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72\" (UID: \"d5355e22-6596-4c63-a492-fe8b578f04b5\") " pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"
Oct 30 01:08:20.731543 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:20.731511    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"
Oct 30 01:08:21.141737 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:08:21.141706    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5355e22_6596_4c63_a492_fe8b578f04b5.slice/crio-e347a81bbbfb67299f435cd0d4a76348b8c48e021b48a347f11925f97b4d0277 WatchSource:0}: Error finding container e347a81bbbfb67299f435cd0d4a76348b8c48e021b48a347f11925f97b4d0277: Status 404 returned error can't find the container with id e347a81bbbfb67299f435cd0d4a76348b8c48e021b48a347f11925f97b4d0277
Oct 30 01:08:21.144163 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:21.143773    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"]
Oct 30 01:08:21.146467 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:21.146424    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:08:21.146596 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:21.146511    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:08:21.146596 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:21.146563    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:08:21.886060 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:21.886024    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72" event={"ID":"d5355e22-6596-4c63-a492-fe8b578f04b5","Type":"ContainerStarted","Data":"12102c58a31179f367c4e9cb084c71ff9b233fd21edbcbe33ad1c0090a4d147e"}
Oct 30 01:08:21.886474 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:21.886088    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72" event={"ID":"d5355e22-6596-4c63-a492-fe8b578f04b5","Type":"ContainerStarted","Data":"e347a81bbbfb67299f435cd0d4a76348b8c48e021b48a347f11925f97b4d0277"}
Oct 30 01:08:21.973174 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:21.973117    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72" podStartSLOduration=1.973102653 podStartE2EDuration="1.973102653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:21.956293425 +0000 UTC m=+2192.343969806" watchObservedRunningTime="2026-10-30 01:08:21.973102653 +0000 UTC m=+2192.360779118"
Oct 30 01:08:22.732484 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:22.732450    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"
Oct 30 01:08:22.734625 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:22.734602    4393 patch_prober.go:29] interesting pod/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 01:08:22.734731 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:22.734650    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72" podUID="d5355e22-6596-4c63-a492-fe8b578f04b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 01:08:23.739390 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:23.739353    4393 patch_prober.go:29] interesting pod/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 01:08:23.740074 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:23.740042    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72" podUID="d5355e22-6596-4c63-a492-fe8b578f04b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 01:08:24.732396 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:24.732354    4393 patch_prober.go:29] interesting pod/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 01:08:24.732596 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:24.732432    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72" podUID="d5355e22-6596-4c63-a492-fe8b578f04b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 01:08:24.886523 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:24.886474    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g"]
Oct 30 01:08:25.736874 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:25.736150    4393 patch_prober.go:29] interesting pod/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 01:08:25.736874 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:25.736211    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72" podUID="d5355e22-6596-4c63-a492-fe8b578f04b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 01:08:26.537374 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:08:26.537332    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:08:26.732852 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:26.732821    4393 patch_prober.go:29] interesting pod/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body=
Oct 30 01:08:26.733034 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:26.732875    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72" podUID="d5355e22-6596-4c63-a492-fe8b578f04b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused"
Oct 30 01:08:27.871129 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:27.871093    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"
Oct 30 01:08:27.883803 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:27.874927    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"
Oct 30 01:08:27.884143 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:27.884119    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"
Oct 30 01:08:29.677404 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:29.676024    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"]
Oct 30 01:08:29.677404 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:29.676357    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn" podUID="0f8bf91a-d2e7-4c6e-90c4-e8d794029f21" containerName="istio-proxy" containerID="cri-o://2530e15d554970bf633648513bbe32e408b1ced3ce4390b170b8593257b80e51" gracePeriod=30
Oct 30 01:08:30.004224 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:30.004142    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn" podUID="73d9db41-7564-4d70-b061-bcb981ca5263" containerName="main" containerID="cri-o://b981c58d0524971abd9d2bb5fe2ff599452cdbc456bc4521bf25edb7ea60930c" gracePeriod=45
Oct 30 01:08:30.293968 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:30.293936    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn"
Oct 30 01:08:30.446008 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:30.445971    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73d9db41-7564-4d70-b061-bcb981ca5263-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73d9db41-7564-4d70-b061-bcb981ca5263-model-cache\") pod \"73d9db41-7564-4d70-b061-bcb981ca5263\" (UID: \"73d9db41-7564-4d70-b061-bcb981ca5263\") "
Oct 30 01:08:30.446205 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:30.446046    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73d9db41-7564-4d70-b061-bcb981ca5263-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73d9db41-7564-4d70-b061-bcb981ca5263-tmp-dir\") pod \"73d9db41-7564-4d70-b061-bcb981ca5263\" (UID: \"73d9db41-7564-4d70-b061-bcb981ca5263\") "
Oct 30 01:08:30.446205 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:30.446171    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73d9db41-7564-4d70-b061-bcb981ca5263-dshm\" (UniqueName: \"kubernetes.io/empty-dir/73d9db41-7564-4d70-b061-bcb981ca5263-dshm\") pod \"73d9db41-7564-4d70-b061-bcb981ca5263\" (UID: \"73d9db41-7564-4d70-b061-bcb981ca5263\") "
Oct 30 01:08:30.446346 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:30.446239    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73d9db41-7564-4d70-b061-bcb981ca5263-kube-api-access-9mpbz\" (UniqueName: \"kubernetes.io/projected/73d9db41-7564-4d70-b061-bcb981ca5263-kube-api-access-9mpbz\") pod \"73d9db41-7564-4d70-b061-bcb981ca5263\" (UID: \"73d9db41-7564-4d70-b061-bcb981ca5263\") "
Oct 30 01:08:30.446346 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:30.446264    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73d9db41-7564-4d70-b061-bcb981ca5263-model-cache" pod "73d9db41-7564-4d70-b061-bcb981ca5263" (UID: "73d9db41-7564-4d70-b061-bcb981ca5263"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:30.446346 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:30.446302    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73d9db41-7564-4d70-b061-bcb981ca5263-tls-certs\" (UniqueName: \"kubernetes.io/secret/73d9db41-7564-4d70-b061-bcb981ca5263-tls-certs\") pod \"73d9db41-7564-4d70-b061-bcb981ca5263\" (UID: \"73d9db41-7564-4d70-b061-bcb981ca5263\") "
Oct 30 01:08:30.446346 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:30.446339    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73d9db41-7564-4d70-b061-bcb981ca5263-home\" (UniqueName: \"kubernetes.io/empty-dir/73d9db41-7564-4d70-b061-bcb981ca5263-home\") pod \"73d9db41-7564-4d70-b061-bcb981ca5263\" (UID: \"73d9db41-7564-4d70-b061-bcb981ca5263\") "
Oct 30 01:08:30.446653 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:30.446359    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73d9db41-7564-4d70-b061-bcb981ca5263-tmp-dir" pod "73d9db41-7564-4d70-b061-bcb981ca5263" (UID: "73d9db41-7564-4d70-b061-bcb981ca5263"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:30.446720 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:30.446700    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73d9db41-7564-4d70-b061-bcb981ca5263-home" pod "73d9db41-7564-4d70-b061-bcb981ca5263" (UID: "73d9db41-7564-4d70-b061-bcb981ca5263"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:30.447110 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:30.447086    4393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73d9db41-7564-4d70-b061-bcb981ca5263-home\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:08:30.447234 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:30.447113    4393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73d9db41-7564-4d70-b061-bcb981ca5263-model-cache\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:08:30.447234 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:30.447127    4393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73d9db41-7564-4d70-b061-bcb981ca5263-tmp-dir\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:08:30.448715 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:30.448691    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73d9db41-7564-4d70-b061-bcb981ca5263-dshm" pod "73d9db41-7564-4d70-b061-bcb981ca5263" (UID: "73d9db41-7564-4d70-b061-bcb981ca5263"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:30.448859 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:30.448833    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73d9db41-7564-4d70-b061-bcb981ca5263-kube-api-access-9mpbz" pod "73d9db41-7564-4d70-b061-bcb981ca5263" (UID: "73d9db41-7564-4d70-b061-bcb981ca5263"). InnerVolumeSpecName "kube-api-access-9mpbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:30.448972 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:30.448952    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73d9db41-7564-4d70-b061-bcb981ca5263-tls-certs" pod "73d9db41-7564-4d70-b061-bcb981ca5263" (UID: "73d9db41-7564-4d70-b061-bcb981ca5263"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:08:30.548507 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:30.548427    4393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73d9db41-7564-4d70-b061-bcb981ca5263-dshm\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:08:30.548507 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:30.548455    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mpbz\" (UniqueName: \"kubernetes.io/projected/73d9db41-7564-4d70-b061-bcb981ca5263-kube-api-access-9mpbz\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:08:30.548507 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:30.548465    4393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73d9db41-7564-4d70-b061-bcb981ca5263-tls-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:08:30.991636 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:30.991595    4393 generic.go:356] "Generic (PLEG): container finished" podID="73d9db41-7564-4d70-b061-bcb981ca5263" containerID="b981c58d0524971abd9d2bb5fe2ff599452cdbc456bc4521bf25edb7ea60930c" exitCode=0
Oct 30 01:08:30.992131 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:30.991647    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn"
Oct 30 01:08:30.992131 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:30.991683    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn" event={"ID":"73d9db41-7564-4d70-b061-bcb981ca5263","Type":"ContainerDied","Data":"b981c58d0524971abd9d2bb5fe2ff599452cdbc456bc4521bf25edb7ea60930c"}
Oct 30 01:08:30.992131 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:30.991726    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn" event={"ID":"73d9db41-7564-4d70-b061-bcb981ca5263","Type":"ContainerDied","Data":"1c97cb727910650982dad691e2c7b23d3d41074bb25089d8bdaff0f6268f4d87"}
Oct 30 01:08:30.992131 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:30.991742    4393 scope.go:122] "RemoveContainer" containerID="b981c58d0524971abd9d2bb5fe2ff599452cdbc456bc4521bf25edb7ea60930c"
Oct 30 01:08:31.004232 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:31.004214    4393 scope.go:122] "RemoveContainer" containerID="b981c58d0524971abd9d2bb5fe2ff599452cdbc456bc4521bf25edb7ea60930c"
Oct 30 01:08:31.004515 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:08:31.004489    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b981c58d0524971abd9d2bb5fe2ff599452cdbc456bc4521bf25edb7ea60930c\": container with ID starting with b981c58d0524971abd9d2bb5fe2ff599452cdbc456bc4521bf25edb7ea60930c not found: ID does not exist" containerID="b981c58d0524971abd9d2bb5fe2ff599452cdbc456bc4521bf25edb7ea60930c"
Oct 30 01:08:31.004595 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:31.004525    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b981c58d0524971abd9d2bb5fe2ff599452cdbc456bc4521bf25edb7ea60930c"} err="failed to get container status \"b981c58d0524971abd9d2bb5fe2ff599452cdbc456bc4521bf25edb7ea60930c\": rpc error: code = NotFound desc = could not find container \"b981c58d0524971abd9d2bb5fe2ff599452cdbc456bc4521bf25edb7ea60930c\": container with ID starting with b981c58d0524971abd9d2bb5fe2ff599452cdbc456bc4521bf25edb7ea60930c not found: ID does not exist"
Oct 30 01:08:31.043693 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:31.043656    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn"]
Oct 30 01:08:31.055225 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:31.055197    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5f387c/auth-test-model-5f387c-kserve-7f7bbc4c89-cp7zn"]
Oct 30 01:08:31.727216 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:31.727179    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73d9db41-7564-4d70-b061-bcb981ca5263" path="/var/lib/kubelet/pods/73d9db41-7564-4d70-b061-bcb981ca5263/volumes"
Oct 30 01:08:35.028053 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:35.028017    4393 generic.go:356] "Generic (PLEG): container finished" podID="0f8bf91a-d2e7-4c6e-90c4-e8d794029f21" containerID="2530e15d554970bf633648513bbe32e408b1ced3ce4390b170b8593257b80e51" exitCode=0
Oct 30 01:08:35.028382 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:35.028082    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn" event={"ID":"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21","Type":"ContainerDied","Data":"2530e15d554970bf633648513bbe32e408b1ced3ce4390b170b8593257b80e51"}
Oct 30 01:08:35.028382 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:35.028130    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn" event={"ID":"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21","Type":"ContainerDied","Data":"474be8e4c93406fd789f049d619cb59a80239a1abb881b71cbb09d9242060ae6"}
Oct 30 01:08:35.028382 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:35.028146    4393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="474be8e4c93406fd789f049d619cb59a80239a1abb881b71cbb09d9242060ae6"
Oct 30 01:08:35.043978 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:35.043941    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"
Oct 30 01:08:35.128056 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:35.128020    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-workload-socket\") pod \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\" (UID: \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\") "
Oct 30 01:08:35.128238 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:35.128115    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-istio-data\") pod \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\" (UID: \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\") "
Oct 30 01:08:35.128238 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:35.128158    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-istio-envoy\") pod \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\" (UID: \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\") "
Oct 30 01:08:35.128238 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:35.128209    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-istio-token\" (UniqueName: \"kubernetes.io/projected/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-istio-token\") pod \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\" (UID: \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\") "
Oct 30 01:08:35.128503 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:35.128353    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-istio-podinfo\") pod \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\" (UID: \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\") "
Oct 30 01:08:35.128503 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:35.128413    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-istio-data" pod "0f8bf91a-d2e7-4c6e-90c4-e8d794029f21" (UID: "0f8bf91a-d2e7-4c6e-90c4-e8d794029f21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:35.128503 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:35.128426    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-workload-socket" pod "0f8bf91a-d2e7-4c6e-90c4-e8d794029f21" (UID: "0f8bf91a-d2e7-4c6e-90c4-e8d794029f21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:35.128503 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:35.128423    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-workload-certs\") pod \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\" (UID: \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\") "
Oct 30 01:08:35.128723 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:35.128503    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-kube-api-access-bzhj6\" (UniqueName: \"kubernetes.io/projected/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-kube-api-access-bzhj6\") pod \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\" (UID: \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\") "
Oct 30 01:08:35.128723 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:35.128624    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-istiod-ca-cert\") pod \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\" (UID: \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\") "
Oct 30 01:08:35.128723 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:35.128662    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-workload-certs" pod "0f8bf91a-d2e7-4c6e-90c4-e8d794029f21" (UID: "0f8bf91a-d2e7-4c6e-90c4-e8d794029f21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:35.128723 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:35.128691    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-credential-socket\") pod \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\" (UID: \"0f8bf91a-d2e7-4c6e-90c4-e8d794029f21\") "
Oct 30 01:08:35.129003 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:35.128980    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-istiod-ca-cert" pod "0f8bf91a-d2e7-4c6e-90c4-e8d794029f21" (UID: "0f8bf91a-d2e7-4c6e-90c4-e8d794029f21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:08:35.129096 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:35.129070    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-credential-socket" pod "0f8bf91a-d2e7-4c6e-90c4-e8d794029f21" (UID: "0f8bf91a-d2e7-4c6e-90c4-e8d794029f21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:35.129651 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:35.129624    4393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-workload-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:08:35.129762 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:35.129656    4393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-istiod-ca-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:08:35.129762 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:35.129671    4393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-credential-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:08:35.129762 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:35.129684    4393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-workload-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:08:35.129762 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:35.129696    4393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-istio-data\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:08:35.131124 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:35.131096    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-istio-envoy" pod "0f8bf91a-d2e7-4c6e-90c4-e8d794029f21" (UID: "0f8bf91a-d2e7-4c6e-90c4-e8d794029f21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:35.131449 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:35.131419    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-istio-podinfo" pod "0f8bf91a-d2e7-4c6e-90c4-e8d794029f21" (UID: "0f8bf91a-d2e7-4c6e-90c4-e8d794029f21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:08:35.131650 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:35.131625    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-kube-api-access-bzhj6" pod "0f8bf91a-d2e7-4c6e-90c4-e8d794029f21" (UID: "0f8bf91a-d2e7-4c6e-90c4-e8d794029f21"). InnerVolumeSpecName "kube-api-access-bzhj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:35.132060 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:35.132035    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-istio-token" pod "0f8bf91a-d2e7-4c6e-90c4-e8d794029f21" (UID: "0f8bf91a-d2e7-4c6e-90c4-e8d794029f21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:35.231462 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:35.231431    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzhj6\" (UniqueName: \"kubernetes.io/projected/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-kube-api-access-bzhj6\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:08:35.231462 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:35.231462    4393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-istio-envoy\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:08:35.231662 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:35.231474    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:08:35.231662 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:35.231484    4393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21-istio-podinfo\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:08:36.036345 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:36.036267    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"
Oct 30 01:08:36.100599 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:36.100565    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"]
Oct 30 01:08:36.104952 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:36.104926    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ef1084-openshift-default-68989c6b9f-jmgcn"]
Oct 30 01:08:37.736955 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:37.736921    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f8bf91a-d2e7-4c6e-90c4-e8d794029f21" path="/var/lib/kubelet/pods/0f8bf91a-d2e7-4c6e-90c4-e8d794029f21/volumes"
Oct 30 01:08:39.943030 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:39.942969    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g" podUID="931977c3-8e00-4f75-b85b-be0666416b82" containerName="main" containerID="cri-o://7ad63c330103adc0cb599c86656b30d66de6e49810027addd507a05a372fadaa" gracePeriod=45
Oct 30 01:08:40.074908 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:40.074877    4393 generic.go:356] "Generic (PLEG): container finished" podID="931977c3-8e00-4f75-b85b-be0666416b82" containerID="7ad63c330103adc0cb599c86656b30d66de6e49810027addd507a05a372fadaa" exitCode=0
Oct 30 01:08:40.075090 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:40.074956    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g" event={"ID":"931977c3-8e00-4f75-b85b-be0666416b82","Type":"ContainerDied","Data":"7ad63c330103adc0cb599c86656b30d66de6e49810027addd507a05a372fadaa"}
Oct 30 01:08:40.297184 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:40.297159    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g"
Oct 30 01:08:40.382032 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:40.381996    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/931977c3-8e00-4f75-b85b-be0666416b82-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/931977c3-8e00-4f75-b85b-be0666416b82-model-cache\") pod \"931977c3-8e00-4f75-b85b-be0666416b82\" (UID: \"931977c3-8e00-4f75-b85b-be0666416b82\") "
Oct 30 01:08:40.382208 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:40.382062    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/931977c3-8e00-4f75-b85b-be0666416b82-kube-api-access-tkwdg\" (UniqueName: \"kubernetes.io/projected/931977c3-8e00-4f75-b85b-be0666416b82-kube-api-access-tkwdg\") pod \"931977c3-8e00-4f75-b85b-be0666416b82\" (UID: \"931977c3-8e00-4f75-b85b-be0666416b82\") "
Oct 30 01:08:40.382208 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:40.382100    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/931977c3-8e00-4f75-b85b-be0666416b82-dshm\" (UniqueName: \"kubernetes.io/empty-dir/931977c3-8e00-4f75-b85b-be0666416b82-dshm\") pod \"931977c3-8e00-4f75-b85b-be0666416b82\" (UID: \"931977c3-8e00-4f75-b85b-be0666416b82\") "
Oct 30 01:08:40.382208 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:40.382151    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/931977c3-8e00-4f75-b85b-be0666416b82-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/931977c3-8e00-4f75-b85b-be0666416b82-tmp-dir\") pod \"931977c3-8e00-4f75-b85b-be0666416b82\" (UID: \"931977c3-8e00-4f75-b85b-be0666416b82\") "
Oct 30 01:08:40.382402 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:40.382288    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/931977c3-8e00-4f75-b85b-be0666416b82-home\" (UniqueName: \"kubernetes.io/empty-dir/931977c3-8e00-4f75-b85b-be0666416b82-home\") pod \"931977c3-8e00-4f75-b85b-be0666416b82\" (UID: \"931977c3-8e00-4f75-b85b-be0666416b82\") "
Oct 30 01:08:40.382402 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:40.382301    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931977c3-8e00-4f75-b85b-be0666416b82-model-cache" pod "931977c3-8e00-4f75-b85b-be0666416b82" (UID: "931977c3-8e00-4f75-b85b-be0666416b82"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:40.382503 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:40.382452    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/931977c3-8e00-4f75-b85b-be0666416b82-tls-certs\" (UniqueName: \"kubernetes.io/secret/931977c3-8e00-4f75-b85b-be0666416b82-tls-certs\") pod \"931977c3-8e00-4f75-b85b-be0666416b82\" (UID: \"931977c3-8e00-4f75-b85b-be0666416b82\") "
Oct 30 01:08:40.382619 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:40.382592    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931977c3-8e00-4f75-b85b-be0666416b82-home" pod "931977c3-8e00-4f75-b85b-be0666416b82" (UID: "931977c3-8e00-4f75-b85b-be0666416b82"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:40.382748 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:40.382727    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931977c3-8e00-4f75-b85b-be0666416b82-tmp-dir" pod "931977c3-8e00-4f75-b85b-be0666416b82" (UID: "931977c3-8e00-4f75-b85b-be0666416b82"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:40.383347 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:40.383318    4393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/931977c3-8e00-4f75-b85b-be0666416b82-model-cache\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:08:40.383347 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:40.383349    4393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/931977c3-8e00-4f75-b85b-be0666416b82-tmp-dir\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:08:40.383531 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:40.383361    4393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/931977c3-8e00-4f75-b85b-be0666416b82-home\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:08:40.384710 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:40.384674    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931977c3-8e00-4f75-b85b-be0666416b82-dshm" pod "931977c3-8e00-4f75-b85b-be0666416b82" (UID: "931977c3-8e00-4f75-b85b-be0666416b82"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:08:40.384993 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:40.384967    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/931977c3-8e00-4f75-b85b-be0666416b82-kube-api-access-tkwdg" pod "931977c3-8e00-4f75-b85b-be0666416b82" (UID: "931977c3-8e00-4f75-b85b-be0666416b82"). InnerVolumeSpecName "kube-api-access-tkwdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:08:40.385329 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:40.385295    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/931977c3-8e00-4f75-b85b-be0666416b82-tls-certs" pod "931977c3-8e00-4f75-b85b-be0666416b82" (UID: "931977c3-8e00-4f75-b85b-be0666416b82"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:08:40.485226 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:40.485187    4393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/931977c3-8e00-4f75-b85b-be0666416b82-dshm\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:08:40.485226 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:40.485218    4393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/931977c3-8e00-4f75-b85b-be0666416b82-tls-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:08:40.485434 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:40.485235    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkwdg\" (UniqueName: \"kubernetes.io/projected/931977c3-8e00-4f75-b85b-be0666416b82-kube-api-access-tkwdg\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:08:41.084867 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:41.084837    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g"
Oct 30 01:08:41.085319 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:41.084845    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g" event={"ID":"931977c3-8e00-4f75-b85b-be0666416b82","Type":"ContainerDied","Data":"bf7ef8fa20842ecee478db91c21d0b2da88d7221d790977a7c7dc77e319a13ea"}
Oct 30 01:08:41.085319 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:41.084984    4393 scope.go:122] "RemoveContainer" containerID="7ad63c330103adc0cb599c86656b30d66de6e49810027addd507a05a372fadaa"
Oct 30 01:08:41.138582 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:41.138543    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g"]
Oct 30 01:08:41.149804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:41.149755    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g"]
Oct 30 01:08:41.196178 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:41.196138    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-00e07a/auth-test-model-00e07a-kserve-594589575c-kd75g" podUID="931977c3-8e00-4f75-b85b-be0666416b82" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:08:41.726932 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:41.726901    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="931977c3-8e00-4f75-b85b-be0666416b82" path="/var/lib/kubelet/pods/931977c3-8e00-4f75-b85b-be0666416b82/volumes"
Oct 30 01:08:54.110423 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.110383    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh"]
Oct 30 01:08:54.111449 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.111425    4393 cpu_manager.go:414] "containerMap: removing container" podUID="0f8bf91a-d2e7-4c6e-90c4-e8d794029f21" containerName="istio-proxy"
Oct 30 01:08:54.111449 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.111449    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f8bf91a-d2e7-4c6e-90c4-e8d794029f21" containerName="istio-proxy"
Oct 30 01:08:54.111640 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.111468    4393 cpu_manager.go:414] "containerMap: removing container" podUID="931977c3-8e00-4f75-b85b-be0666416b82" containerName="main"
Oct 30 01:08:54.111640 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.111476    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="931977c3-8e00-4f75-b85b-be0666416b82" containerName="main"
Oct 30 01:08:54.111640 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.111490    4393 cpu_manager.go:414] "containerMap: removing container" podUID="73d9db41-7564-4d70-b061-bcb981ca5263" containerName="main"
Oct 30 01:08:54.111640 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.111497    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73d9db41-7564-4d70-b061-bcb981ca5263" containerName="main"
Oct 30 01:08:54.111882 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.111706    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f8bf91a-d2e7-4c6e-90c4-e8d794029f21" containerName="istio-proxy"
Oct 30 01:08:54.111882 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.111725    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="931977c3-8e00-4f75-b85b-be0666416b82" containerName="main"
Oct 30 01:08:54.111882 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.111736    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="73d9db41-7564-4d70-b061-bcb981ca5263" containerName="main"
Oct 30 01:08:54.117132 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.117109    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh"
Oct 30 01:08:54.135186 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.135153    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh"]
Oct 30 01:08:54.149824 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.141910    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-a5e9ab-serving-cert\""
Oct 30 01:08:54.245271 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.245228    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh\" (UID: \"6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh"
Oct 30 01:08:54.245447 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.245285    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jncfh\" (UniqueName: \"kubernetes.io/projected/6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4-kube-api-access-jncfh\") pod \"maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh\" (UID: \"6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh"
Oct 30 01:08:54.245647 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.245627    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh\" (UID: \"6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh"
Oct 30 01:08:54.245752 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.245730    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh\" (UID: \"6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh"
Oct 30 01:08:54.346961 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.346928    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh\" (UID: \"6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh"
Oct 30 01:08:54.347171 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.347004    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh\" (UID: \"6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh"
Oct 30 01:08:54.347171 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.347085    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh\" (UID: \"6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh"
Oct 30 01:08:54.347171 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.347114    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jncfh\" (UniqueName: \"kubernetes.io/projected/6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4-kube-api-access-jncfh\") pod \"maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh\" (UID: \"6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh"
Oct 30 01:08:54.347714 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.347688    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh\" (UID: \"6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh"
Oct 30 01:08:54.349674 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.349648    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh\" (UID: \"6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh"
Oct 30 01:08:54.349841 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.349817    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh\" (UID: \"6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh"
Oct 30 01:08:54.369717 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.369685    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jncfh\" (UniqueName: \"kubernetes.io/projected/6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4-kube-api-access-jncfh\") pod \"maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh\" (UID: \"6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh"
Oct 30 01:08:54.448802 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.442641    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh"
Oct 30 01:08:54.767019 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:08:54.766985    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6634f2ef_e6cc_4dc2_bc18_55cf33bc9cd4.slice/crio-358b354d57aa8955adadc7ccc04f469bd86c5d01aa54440b9e05ee39b6863b97 WatchSource:0}: Error finding container 358b354d57aa8955adadc7ccc04f469bd86c5d01aa54440b9e05ee39b6863b97: Status 404 returned error can't find the container with id 358b354d57aa8955adadc7ccc04f469bd86c5d01aa54440b9e05ee39b6863b97
Oct 30 01:08:54.767551 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.767494    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh"]
Oct 30 01:08:54.850976 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.850939    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7"]
Oct 30 01:08:54.860142 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.860110    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7"
Oct 30 01:08:54.876574 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.876548    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7"]
Oct 30 01:08:54.883511 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.883045    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-7f92f6\"/\"auth-test-model-7f92f6-kserve-self-signed-certs\""
Oct 30 01:08:54.883511 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.883045    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-7f92f6\"/\"openshift-service-ca.crt\""
Oct 30 01:08:54.883511 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.883068    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-7f92f6\"/\"kube-root-ca.crt\""
Oct 30 01:08:54.883511 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.883082    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-7f92f6\"/\"default-dockercfg-drc54\""
Oct 30 01:08:54.957305 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.957271    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-dshm\") pod \"auth-test-model-7f92f6-kserve-5d69666869-t2hc7\" (UID: \"ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4\") " pod="ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7"
Oct 30 01:08:54.957305 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.957303    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-home\") pod \"auth-test-model-7f92f6-kserve-5d69666869-t2hc7\" (UID: \"ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4\") " pod="ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7"
Oct 30 01:08:54.957511 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.957336    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v72lr\" (UniqueName: \"kubernetes.io/projected/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-kube-api-access-v72lr\") pod \"auth-test-model-7f92f6-kserve-5d69666869-t2hc7\" (UID: \"ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4\") " pod="ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7"
Oct 30 01:08:54.957511 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.957436    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-tmp-dir\") pod \"auth-test-model-7f92f6-kserve-5d69666869-t2hc7\" (UID: \"ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4\") " pod="ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7"
Oct 30 01:08:54.957643 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.957624    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-tls-certs\") pod \"auth-test-model-7f92f6-kserve-5d69666869-t2hc7\" (UID: \"ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4\") " pod="ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7"
Oct 30 01:08:54.957685 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:54.957670    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-model-cache\") pod \"auth-test-model-7f92f6-kserve-5d69666869-t2hc7\" (UID: \"ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4\") " pod="ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7"
Oct 30 01:08:55.059180 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:55.059101    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-tls-certs\") pod \"auth-test-model-7f92f6-kserve-5d69666869-t2hc7\" (UID: \"ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4\") " pod="ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7"
Oct 30 01:08:55.059180 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:55.059157    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-model-cache\") pod \"auth-test-model-7f92f6-kserve-5d69666869-t2hc7\" (UID: \"ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4\") " pod="ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7"
Oct 30 01:08:55.059419 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:55.059289    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-dshm\") pod \"auth-test-model-7f92f6-kserve-5d69666869-t2hc7\" (UID: \"ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4\") " pod="ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7"
Oct 30 01:08:55.059419 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:55.059308    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-home\") pod \"auth-test-model-7f92f6-kserve-5d69666869-t2hc7\" (UID: \"ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4\") " pod="ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7"
Oct 30 01:08:55.059419 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:55.059337    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v72lr\" (UniqueName: \"kubernetes.io/projected/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-kube-api-access-v72lr\") pod \"auth-test-model-7f92f6-kserve-5d69666869-t2hc7\" (UID: \"ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4\") " pod="ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7"
Oct 30 01:08:55.059419 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:55.059386    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-tmp-dir\") pod \"auth-test-model-7f92f6-kserve-5d69666869-t2hc7\" (UID: \"ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4\") " pod="ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7"
Oct 30 01:08:55.059645 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:55.059605    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-model-cache\") pod \"auth-test-model-7f92f6-kserve-5d69666869-t2hc7\" (UID: \"ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4\") " pod="ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7"
Oct 30 01:08:55.060004 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:55.059880    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-tmp-dir\") pod \"auth-test-model-7f92f6-kserve-5d69666869-t2hc7\" (UID: \"ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4\") " pod="ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7"
Oct 30 01:08:55.060004 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:55.059897    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-home\") pod \"auth-test-model-7f92f6-kserve-5d69666869-t2hc7\" (UID: \"ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4\") " pod="ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7"
Oct 30 01:08:55.061758 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:55.061732    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-dshm\") pod \"auth-test-model-7f92f6-kserve-5d69666869-t2hc7\" (UID: \"ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4\") " pod="ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7"
Oct 30 01:08:55.061956 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:55.061937    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-tls-certs\") pod \"auth-test-model-7f92f6-kserve-5d69666869-t2hc7\" (UID: \"ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4\") " pod="ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7"
Oct 30 01:08:55.104041 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:55.103015    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v72lr\" (UniqueName: \"kubernetes.io/projected/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-kube-api-access-v72lr\") pod \"auth-test-model-7f92f6-kserve-5d69666869-t2hc7\" (UID: \"ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4\") " pod="ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7"
Oct 30 01:08:55.186130 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:55.186097    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7"
Oct 30 01:08:55.240680 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:55.240646    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh" event={"ID":"6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4","Type":"ContainerStarted","Data":"358b354d57aa8955adadc7ccc04f469bd86c5d01aa54440b9e05ee39b6863b97"}
Oct 30 01:08:55.553826 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:08:55.542888    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca5cbcaf_46cc_4a51_b372_10a6fd84f2f4.slice/crio-588dd7c38b70c376a1bbcfb1c135234ebc765e583a10e918f350136d127d8d1c WatchSource:0}: Error finding container 588dd7c38b70c376a1bbcfb1c135234ebc765e583a10e918f350136d127d8d1c: Status 404 returned error can't find the container with id 588dd7c38b70c376a1bbcfb1c135234ebc765e583a10e918f350136d127d8d1c
Oct 30 01:08:55.589394 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:55.588449    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7"]
Oct 30 01:08:56.252090 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:56.252056    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh" event={"ID":"6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4","Type":"ContainerStarted","Data":"65ed09760d48038dd2dd794ff911eb32cf15564a1a8a1393f70737ce1ebb7cd9"}
Oct 30 01:08:56.252590 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:56.252140    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh"
Oct 30 01:08:56.253894 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:56.253868    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7" event={"ID":"ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4","Type":"ContainerStarted","Data":"0b798ce7e605dbcbd34c34afc1ee35d32f984e1e864f5166b89165c12bdb3cbe"}
Oct 30 01:08:56.254030 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:56.253905    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7" event={"ID":"ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4","Type":"ContainerStarted","Data":"588dd7c38b70c376a1bbcfb1c135234ebc765e583a10e918f350136d127d8d1c"}
Oct 30 01:08:56.307804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:56.299306    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh" podStartSLOduration=1.924370713 podStartE2EDuration="2.299285426s" totalImagesPullingTime="374.914713ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:56.294055657 +0000 UTC m=+2226.681732159" watchObservedRunningTime="2026-10-30 01:08:56.299285426 +0000 UTC m=+2226.686961818"
Oct 30 01:08:56.357848 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:56.353333    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7" podStartSLOduration=2.353311726 podStartE2EDuration="2.353311726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:08:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:08:56.342205774 +0000 UTC m=+2226.729882176" watchObservedRunningTime="2026-10-30 01:08:56.353311726 +0000 UTC m=+2226.740988115"
Oct 30 01:08:56.526510 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:08:56.526422    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:08:57.186845 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:08:57.186806    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7"
Oct 30 01:09:02.266763 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:02.266730    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh"
Oct 30 01:09:04.209795 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:04.200859    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9"]
Oct 30 01:09:04.482252 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:04.482171    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm"]
Oct 30 01:09:04.482435 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:04.482414    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm" podUID="77816148-3d88-40c8-8ace-5a71f39dd378" containerName="maas-api" containerID="cri-o://694cc7ad9c7fa40e7307101fc3c2ccc8ce3865bcb00c357acb1e7442bc550603" gracePeriod=30
Oct 30 01:09:04.881368 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:04.881330    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm"
Oct 30 01:09:05.106582 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:05.106408    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77816148-3d88-40c8-8ace-5a71f39dd378-metrics-certs\" (UniqueName: \"kubernetes.io/secret/77816148-3d88-40c8-8ace-5a71f39dd378-metrics-certs\") pod \"77816148-3d88-40c8-8ace-5a71f39dd378\" (UID: \"77816148-3d88-40c8-8ace-5a71f39dd378\") "
Oct 30 01:09:05.106582 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:05.106537    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77816148-3d88-40c8-8ace-5a71f39dd378-kube-api-access-v9ndl\" (UniqueName: \"kubernetes.io/projected/77816148-3d88-40c8-8ace-5a71f39dd378-kube-api-access-v9ndl\") pod \"77816148-3d88-40c8-8ace-5a71f39dd378\" (UID: \"77816148-3d88-40c8-8ace-5a71f39dd378\") "
Oct 30 01:09:05.106845 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:05.106812    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77816148-3d88-40c8-8ace-5a71f39dd378-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77816148-3d88-40c8-8ace-5a71f39dd378-openshift-service-ca\") pod \"77816148-3d88-40c8-8ace-5a71f39dd378\" (UID: \"77816148-3d88-40c8-8ace-5a71f39dd378\") "
Oct 30 01:09:05.106953 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:05.106934    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77816148-3d88-40c8-8ace-5a71f39dd378-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77816148-3d88-40c8-8ace-5a71f39dd378-maas-api-tls\") pod \"77816148-3d88-40c8-8ace-5a71f39dd378\" (UID: \"77816148-3d88-40c8-8ace-5a71f39dd378\") "
Oct 30 01:09:05.107914 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:05.107885    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77816148-3d88-40c8-8ace-5a71f39dd378-openshift-service-ca" pod "77816148-3d88-40c8-8ace-5a71f39dd378" (UID: "77816148-3d88-40c8-8ace-5a71f39dd378"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:05.111197 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:05.111173    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77816148-3d88-40c8-8ace-5a71f39dd378-maas-api-tls" pod "77816148-3d88-40c8-8ace-5a71f39dd378" (UID: "77816148-3d88-40c8-8ace-5a71f39dd378"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:09:05.111299 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:05.111204    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77816148-3d88-40c8-8ace-5a71f39dd378-metrics-certs" pod "77816148-3d88-40c8-8ace-5a71f39dd378" (UID: "77816148-3d88-40c8-8ace-5a71f39dd378"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:09:05.111299 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:05.111234    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77816148-3d88-40c8-8ace-5a71f39dd378-kube-api-access-v9ndl" pod "77816148-3d88-40c8-8ace-5a71f39dd378" (UID: "77816148-3d88-40c8-8ace-5a71f39dd378"). InnerVolumeSpecName "kube-api-access-v9ndl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:05.187544 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:05.187509    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7"
Oct 30 01:09:05.203184 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:05.203154    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7"
Oct 30 01:09:05.209109 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:05.209076    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77816148-3d88-40c8-8ace-5a71f39dd378-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:09:05.209286 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:05.209115    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77816148-3d88-40c8-8ace-5a71f39dd378-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:09:05.209286 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:05.209133    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77816148-3d88-40c8-8ace-5a71f39dd378-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:09:05.209286 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:05.209147    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9ndl\" (UniqueName: \"kubernetes.io/projected/77816148-3d88-40c8-8ace-5a71f39dd378-kube-api-access-v9ndl\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:09:05.381948 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:05.381908    4393 generic.go:356] "Generic (PLEG): container finished" podID="77816148-3d88-40c8-8ace-5a71f39dd378" containerID="694cc7ad9c7fa40e7307101fc3c2ccc8ce3865bcb00c357acb1e7442bc550603" exitCode=0
Oct 30 01:09:05.382386 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:05.381958    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm"
Oct 30 01:09:05.382386 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:05.381986    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm" event={"ID":"77816148-3d88-40c8-8ace-5a71f39dd378","Type":"ContainerDied","Data":"694cc7ad9c7fa40e7307101fc3c2ccc8ce3865bcb00c357acb1e7442bc550603"}
Oct 30 01:09:05.382386 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:05.382039    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm" event={"ID":"77816148-3d88-40c8-8ace-5a71f39dd378","Type":"ContainerDied","Data":"cdfce79e4f891aa8d693ceb523fa9f432d27c52c7021d60f1eaf56b3668b90b2"}
Oct 30 01:09:05.382386 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:05.382063    4393 scope.go:122] "RemoveContainer" containerID="694cc7ad9c7fa40e7307101fc3c2ccc8ce3865bcb00c357acb1e7442bc550603"
Oct 30 01:09:05.398354 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:05.398332    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7"
Oct 30 01:09:05.400328 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:05.400310    4393 scope.go:122] "RemoveContainer" containerID="694cc7ad9c7fa40e7307101fc3c2ccc8ce3865bcb00c357acb1e7442bc550603"
Oct 30 01:09:05.400646 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:09:05.400624    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"694cc7ad9c7fa40e7307101fc3c2ccc8ce3865bcb00c357acb1e7442bc550603\": container with ID starting with 694cc7ad9c7fa40e7307101fc3c2ccc8ce3865bcb00c357acb1e7442bc550603 not found: ID does not exist" containerID="694cc7ad9c7fa40e7307101fc3c2ccc8ce3865bcb00c357acb1e7442bc550603"
Oct 30 01:09:05.400717 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:05.400654    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"694cc7ad9c7fa40e7307101fc3c2ccc8ce3865bcb00c357acb1e7442bc550603"} err="failed to get container status \"694cc7ad9c7fa40e7307101fc3c2ccc8ce3865bcb00c357acb1e7442bc550603\": rpc error: code = NotFound desc = could not find container \"694cc7ad9c7fa40e7307101fc3c2ccc8ce3865bcb00c357acb1e7442bc550603\": container with ID starting with 694cc7ad9c7fa40e7307101fc3c2ccc8ce3865bcb00c357acb1e7442bc550603 not found: ID does not exist"
Oct 30 01:09:05.438823 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:05.438786    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm"]
Oct 30 01:09:05.443699 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:05.443663    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-19bc68-6947ddd745-8zmcm"]
Oct 30 01:09:05.772598 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:05.772514    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77816148-3d88-40c8-8ace-5a71f39dd378" path="/var/lib/kubelet/pods/77816148-3d88-40c8-8ace-5a71f39dd378/volumes"
Oct 30 01:09:09.125298 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:09.125263    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4"]
Oct 30 01:09:09.126354 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:09.126330    4393 cpu_manager.go:414] "containerMap: removing container" podUID="77816148-3d88-40c8-8ace-5a71f39dd378" containerName="maas-api"
Oct 30 01:09:09.126354 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:09.126352    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77816148-3d88-40c8-8ace-5a71f39dd378" containerName="maas-api"
Oct 30 01:09:09.126571 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:09.126558    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="77816148-3d88-40c8-8ace-5a71f39dd378" containerName="maas-api"
Oct 30 01:09:09.131982 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:09.131963    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4"
Oct 30 01:09:09.153278 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:09.153240    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4"]
Oct 30 01:09:09.166587 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:09.166537    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-55a4c7-serving-cert\""
Oct 30 01:09:09.189754 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:09.189717    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/734513f7-51fc-49f0-98a4-bf801eb09fb1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4\" (UID: \"734513f7-51fc-49f0-98a4-bf801eb09fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4"
Oct 30 01:09:09.190026 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:09.190007    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2wlx\" (UniqueName: \"kubernetes.io/projected/734513f7-51fc-49f0-98a4-bf801eb09fb1-kube-api-access-w2wlx\") pod \"maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4\" (UID: \"734513f7-51fc-49f0-98a4-bf801eb09fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4"
Oct 30 01:09:09.190285 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:09.190255    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/734513f7-51fc-49f0-98a4-bf801eb09fb1-metrics-certs\") pod \"maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4\" (UID: \"734513f7-51fc-49f0-98a4-bf801eb09fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4"
Oct 30 01:09:09.190585 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:09.190563    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/734513f7-51fc-49f0-98a4-bf801eb09fb1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4\" (UID: \"734513f7-51fc-49f0-98a4-bf801eb09fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4"
Oct 30 01:09:09.292547 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:09.292492    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/734513f7-51fc-49f0-98a4-bf801eb09fb1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4\" (UID: \"734513f7-51fc-49f0-98a4-bf801eb09fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4"
Oct 30 01:09:09.292731 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:09.292704    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/734513f7-51fc-49f0-98a4-bf801eb09fb1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4\" (UID: \"734513f7-51fc-49f0-98a4-bf801eb09fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4"
Oct 30 01:09:09.292849 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:09.292759    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2wlx\" (UniqueName: \"kubernetes.io/projected/734513f7-51fc-49f0-98a4-bf801eb09fb1-kube-api-access-w2wlx\") pod \"maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4\" (UID: \"734513f7-51fc-49f0-98a4-bf801eb09fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4"
Oct 30 01:09:09.292917 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:09.292885    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/734513f7-51fc-49f0-98a4-bf801eb09fb1-metrics-certs\") pod \"maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4\" (UID: \"734513f7-51fc-49f0-98a4-bf801eb09fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4"
Oct 30 01:09:09.294323 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:09.294300    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/734513f7-51fc-49f0-98a4-bf801eb09fb1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4\" (UID: \"734513f7-51fc-49f0-98a4-bf801eb09fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4"
Oct 30 01:09:09.297029 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:09.296997    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/734513f7-51fc-49f0-98a4-bf801eb09fb1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4\" (UID: \"734513f7-51fc-49f0-98a4-bf801eb09fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4"
Oct 30 01:09:09.297878 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:09.297851    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/734513f7-51fc-49f0-98a4-bf801eb09fb1-metrics-certs\") pod \"maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4\" (UID: \"734513f7-51fc-49f0-98a4-bf801eb09fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4"
Oct 30 01:09:09.317545 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:09.317521    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2wlx\" (UniqueName: \"kubernetes.io/projected/734513f7-51fc-49f0-98a4-bf801eb09fb1-kube-api-access-w2wlx\") pod \"maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4\" (UID: \"734513f7-51fc-49f0-98a4-bf801eb09fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4"
Oct 30 01:09:09.490855 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:09.483385    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4"
Oct 30 01:09:09.905607 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:09.905580    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4"]
Oct 30 01:09:09.907083 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:09:09.907055    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod734513f7_51fc_49f0_98a4_bf801eb09fb1.slice/crio-f8ded8b16aae98f85fa386bd43f6ddfb52773e30341941c266d4e0b83cac35da WatchSource:0}: Error finding container f8ded8b16aae98f85fa386bd43f6ddfb52773e30341941c266d4e0b83cac35da: Status 404 returned error can't find the container with id f8ded8b16aae98f85fa386bd43f6ddfb52773e30341941c266d4e0b83cac35da
Oct 30 01:09:10.431557 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:10.431519    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4" event={"ID":"734513f7-51fc-49f0-98a4-bf801eb09fb1","Type":"ContainerStarted","Data":"7097715569e230b160ade9a20ca375c64c8ec117f1cd65d9832406ca8c125d46"}
Oct 30 01:09:10.431972 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:10.431580    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4" event={"ID":"734513f7-51fc-49f0-98a4-bf801eb09fb1","Type":"ContainerStarted","Data":"f8ded8b16aae98f85fa386bd43f6ddfb52773e30341941c266d4e0b83cac35da"}
Oct 30 01:09:10.431972 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:10.431601    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4"
Oct 30 01:09:10.479525 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:10.479430    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4" podStartSLOduration=1.180142926 podStartE2EDuration="1.479414967s" totalImagesPullingTime="299.272041ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:10.474585549 +0000 UTC m=+2240.862261935" watchObservedRunningTime="2026-10-30 01:09:10.479414967 +0000 UTC m=+2240.867091375"
Oct 30 01:09:16.459389 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:16.459355    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4"
Oct 30 01:09:19.227447 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:19.227392    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9" podUID="80640906-b954-482d-bec9-4ff13cd07180" containerName="main" containerID="cri-o://b20c60eaa0c06ecb158788eab3299b093710fe807d05dbd6eae9bf00240a57f9" gracePeriod=45
Oct 30 01:09:19.347614 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:19.347237    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9" podUID="80640906-b954-482d-bec9-4ff13cd07180" containerName="main" probeResult="failure" output="Get \"https://10.133.0.154:8000/ready\": dial tcp 10.133.0.154:8000: connect: connection refused"
Oct 30 01:09:19.541662 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:19.541579    4393 generic.go:356] "Generic (PLEG): container finished" podID="80640906-b954-482d-bec9-4ff13cd07180" containerID="b20c60eaa0c06ecb158788eab3299b093710fe807d05dbd6eae9bf00240a57f9" exitCode=0
Oct 30 01:09:19.541868 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:19.541658    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9" event={"ID":"80640906-b954-482d-bec9-4ff13cd07180","Type":"ContainerDied","Data":"b20c60eaa0c06ecb158788eab3299b093710fe807d05dbd6eae9bf00240a57f9"}
Oct 30 01:09:19.582002 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:19.581974    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9"
Oct 30 01:09:19.610217 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:19.610181    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80640906-b954-482d-bec9-4ff13cd07180-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80640906-b954-482d-bec9-4ff13cd07180-tmp-dir\") pod \"80640906-b954-482d-bec9-4ff13cd07180\" (UID: \"80640906-b954-482d-bec9-4ff13cd07180\") "
Oct 30 01:09:19.610410 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:19.610247    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80640906-b954-482d-bec9-4ff13cd07180-home\" (UniqueName: \"kubernetes.io/empty-dir/80640906-b954-482d-bec9-4ff13cd07180-home\") pod \"80640906-b954-482d-bec9-4ff13cd07180\" (UID: \"80640906-b954-482d-bec9-4ff13cd07180\") "
Oct 30 01:09:19.610410 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:19.610355    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80640906-b954-482d-bec9-4ff13cd07180-dshm\" (UniqueName: \"kubernetes.io/empty-dir/80640906-b954-482d-bec9-4ff13cd07180-dshm\") pod \"80640906-b954-482d-bec9-4ff13cd07180\" (UID: \"80640906-b954-482d-bec9-4ff13cd07180\") "
Oct 30 01:09:19.610528 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:19.610413    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80640906-b954-482d-bec9-4ff13cd07180-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80640906-b954-482d-bec9-4ff13cd07180-model-cache\") pod \"80640906-b954-482d-bec9-4ff13cd07180\" (UID: \"80640906-b954-482d-bec9-4ff13cd07180\") "
Oct 30 01:09:19.610528 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:19.610449    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80640906-b954-482d-bec9-4ff13cd07180-tls-certs\" (UniqueName: \"kubernetes.io/secret/80640906-b954-482d-bec9-4ff13cd07180-tls-certs\") pod \"80640906-b954-482d-bec9-4ff13cd07180\" (UID: \"80640906-b954-482d-bec9-4ff13cd07180\") "
Oct 30 01:09:19.610528 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:19.610505    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80640906-b954-482d-bec9-4ff13cd07180-home" pod "80640906-b954-482d-bec9-4ff13cd07180" (UID: "80640906-b954-482d-bec9-4ff13cd07180"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:19.610698 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:19.610516    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80640906-b954-482d-bec9-4ff13cd07180-tmp-dir" pod "80640906-b954-482d-bec9-4ff13cd07180" (UID: "80640906-b954-482d-bec9-4ff13cd07180"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:19.610698 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:19.610547    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80640906-b954-482d-bec9-4ff13cd07180-kube-api-access-q6w85\" (UniqueName: \"kubernetes.io/projected/80640906-b954-482d-bec9-4ff13cd07180-kube-api-access-q6w85\") pod \"80640906-b954-482d-bec9-4ff13cd07180\" (UID: \"80640906-b954-482d-bec9-4ff13cd07180\") "
Oct 30 01:09:19.610834 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:19.610727    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80640906-b954-482d-bec9-4ff13cd07180-model-cache" pod "80640906-b954-482d-bec9-4ff13cd07180" (UID: "80640906-b954-482d-bec9-4ff13cd07180"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:19.611596 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:19.611570    4393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80640906-b954-482d-bec9-4ff13cd07180-model-cache\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:09:19.611728 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:19.611599    4393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80640906-b954-482d-bec9-4ff13cd07180-tmp-dir\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:09:19.611728 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:19.611613    4393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80640906-b954-482d-bec9-4ff13cd07180-home\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:09:19.613043 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:19.613010    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80640906-b954-482d-bec9-4ff13cd07180-tls-certs" pod "80640906-b954-482d-bec9-4ff13cd07180" (UID: "80640906-b954-482d-bec9-4ff13cd07180"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:09:19.613146 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:19.613102    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80640906-b954-482d-bec9-4ff13cd07180-kube-api-access-q6w85" pod "80640906-b954-482d-bec9-4ff13cd07180" (UID: "80640906-b954-482d-bec9-4ff13cd07180"). InnerVolumeSpecName "kube-api-access-q6w85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:19.613146 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:19.613123    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80640906-b954-482d-bec9-4ff13cd07180-dshm" pod "80640906-b954-482d-bec9-4ff13cd07180" (UID: "80640906-b954-482d-bec9-4ff13cd07180"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:19.712894 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:19.712867    4393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80640906-b954-482d-bec9-4ff13cd07180-dshm\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:09:19.712894 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:19.712892    4393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80640906-b954-482d-bec9-4ff13cd07180-tls-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:09:19.713099 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:19.712903    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6w85\" (UniqueName: \"kubernetes.io/projected/80640906-b954-482d-bec9-4ff13cd07180-kube-api-access-q6w85\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:09:20.553000 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:20.552968    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9"
Oct 30 01:09:20.553470 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:20.552975    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9" event={"ID":"80640906-b954-482d-bec9-4ff13cd07180","Type":"ContainerDied","Data":"e08888ed8611aebb2b4101ddf6d393bca4d12bf8254a759fa93f8195d5efaeaa"}
Oct 30 01:09:20.553470 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:20.553098    4393 scope.go:122] "RemoveContainer" containerID="b20c60eaa0c06ecb158788eab3299b093710fe807d05dbd6eae9bf00240a57f9"
Oct 30 01:09:20.604642 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:20.604611    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9"]
Oct 30 01:09:20.616263 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:20.616236    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-cce513/rate-test-model-cce513-kserve-5577645597-xgpn9"]
Oct 30 01:09:21.726392 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:21.726358    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80640906-b954-482d-bec9-4ff13cd07180" path="/var/lib/kubelet/pods/80640906-b954-482d-bec9-4ff13cd07180/volumes"
Oct 30 01:09:26.512014 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:09:26.511968    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:09:39.729616 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:39.729581    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf"]
Oct 30 01:09:39.730569 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:39.730538    4393 cpu_manager.go:414] "containerMap: removing container" podUID="80640906-b954-482d-bec9-4ff13cd07180" containerName="main"
Oct 30 01:09:39.730569 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:39.730569    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80640906-b954-482d-bec9-4ff13cd07180" containerName="main"
Oct 30 01:09:39.730804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:39.730753    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="80640906-b954-482d-bec9-4ff13cd07180" containerName="main"
Oct 30 01:09:39.736500 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:39.736478    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf"
Oct 30 01:09:39.752843 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:39.752812    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf"]
Oct 30 01:09:39.755716 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:39.755691    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-93079c\"/\"kube-root-ca.crt\""
Oct 30 01:09:39.755716 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:39.755693    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-93079c\"/\"openshift-service-ca.crt\""
Oct 30 01:09:39.769837 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:39.762951    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-93079c\"/\"default-dockercfg-pjwr6\""
Oct 30 01:09:39.769837 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:39.763064    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-93079c\"/\"auth-test-model-93079c-kserve-self-signed-certs\""
Oct 30 01:09:39.868532 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:39.868500    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7a01fa7-260e-466c-b416-aad24c892577-tls-certs\") pod \"auth-test-model-93079c-kserve-775c79c645-55zgf\" (UID: \"d7a01fa7-260e-466c-b416-aad24c892577\") " pod="ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf"
Oct 30 01:09:39.868734 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:39.868591    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7a01fa7-260e-466c-b416-aad24c892577-tmp-dir\") pod \"auth-test-model-93079c-kserve-775c79c645-55zgf\" (UID: \"d7a01fa7-260e-466c-b416-aad24c892577\") " pod="ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf"
Oct 30 01:09:39.868917 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:39.868893    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7a01fa7-260e-466c-b416-aad24c892577-model-cache\") pod \"auth-test-model-93079c-kserve-775c79c645-55zgf\" (UID: \"d7a01fa7-260e-466c-b416-aad24c892577\") " pod="ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf"
Oct 30 01:09:39.869048 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:39.869029    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztxrv\" (UniqueName: \"kubernetes.io/projected/d7a01fa7-260e-466c-b416-aad24c892577-kube-api-access-ztxrv\") pod \"auth-test-model-93079c-kserve-775c79c645-55zgf\" (UID: \"d7a01fa7-260e-466c-b416-aad24c892577\") " pod="ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf"
Oct 30 01:09:39.869124 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:39.869106    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7a01fa7-260e-466c-b416-aad24c892577-dshm\") pod \"auth-test-model-93079c-kserve-775c79c645-55zgf\" (UID: \"d7a01fa7-260e-466c-b416-aad24c892577\") " pod="ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf"
Oct 30 01:09:39.869189 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:39.869179    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7a01fa7-260e-466c-b416-aad24c892577-home\") pod \"auth-test-model-93079c-kserve-775c79c645-55zgf\" (UID: \"d7a01fa7-260e-466c-b416-aad24c892577\") " pod="ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf"
Oct 30 01:09:39.992893 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:39.992169    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7a01fa7-260e-466c-b416-aad24c892577-model-cache\") pod \"auth-test-model-93079c-kserve-775c79c645-55zgf\" (UID: \"d7a01fa7-260e-466c-b416-aad24c892577\") " pod="ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf"
Oct 30 01:09:39.992893 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:39.992286    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztxrv\" (UniqueName: \"kubernetes.io/projected/d7a01fa7-260e-466c-b416-aad24c892577-kube-api-access-ztxrv\") pod \"auth-test-model-93079c-kserve-775c79c645-55zgf\" (UID: \"d7a01fa7-260e-466c-b416-aad24c892577\") " pod="ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf"
Oct 30 01:09:39.992893 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:39.992320    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7a01fa7-260e-466c-b416-aad24c892577-dshm\") pod \"auth-test-model-93079c-kserve-775c79c645-55zgf\" (UID: \"d7a01fa7-260e-466c-b416-aad24c892577\") " pod="ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf"
Oct 30 01:09:39.992893 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:39.992418    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7a01fa7-260e-466c-b416-aad24c892577-home\") pod \"auth-test-model-93079c-kserve-775c79c645-55zgf\" (UID: \"d7a01fa7-260e-466c-b416-aad24c892577\") " pod="ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf"
Oct 30 01:09:39.992893 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:39.992609    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7a01fa7-260e-466c-b416-aad24c892577-tls-certs\") pod \"auth-test-model-93079c-kserve-775c79c645-55zgf\" (UID: \"d7a01fa7-260e-466c-b416-aad24c892577\") " pod="ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf"
Oct 30 01:09:39.992893 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:39.992673    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7a01fa7-260e-466c-b416-aad24c892577-tmp-dir\") pod \"auth-test-model-93079c-kserve-775c79c645-55zgf\" (UID: \"d7a01fa7-260e-466c-b416-aad24c892577\") " pod="ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf"
Oct 30 01:09:39.997803 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:39.993769    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7a01fa7-260e-466c-b416-aad24c892577-home\") pod \"auth-test-model-93079c-kserve-775c79c645-55zgf\" (UID: \"d7a01fa7-260e-466c-b416-aad24c892577\") " pod="ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf"
Oct 30 01:09:40.002803 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:39.998947    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7a01fa7-260e-466c-b416-aad24c892577-model-cache\") pod \"auth-test-model-93079c-kserve-775c79c645-55zgf\" (UID: \"d7a01fa7-260e-466c-b416-aad24c892577\") " pod="ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf"
Oct 30 01:09:40.002803 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:39.999487    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7a01fa7-260e-466c-b416-aad24c892577-tmp-dir\") pod \"auth-test-model-93079c-kserve-775c79c645-55zgf\" (UID: \"d7a01fa7-260e-466c-b416-aad24c892577\") " pod="ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf"
Oct 30 01:09:40.020802 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:40.011153    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7a01fa7-260e-466c-b416-aad24c892577-dshm\") pod \"auth-test-model-93079c-kserve-775c79c645-55zgf\" (UID: \"d7a01fa7-260e-466c-b416-aad24c892577\") " pod="ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf"
Oct 30 01:09:40.041923 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:40.041892    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7a01fa7-260e-466c-b416-aad24c892577-tls-certs\") pod \"auth-test-model-93079c-kserve-775c79c645-55zgf\" (UID: \"d7a01fa7-260e-466c-b416-aad24c892577\") " pod="ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf"
Oct 30 01:09:40.063848 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:40.060900    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztxrv\" (UniqueName: \"kubernetes.io/projected/d7a01fa7-260e-466c-b416-aad24c892577-kube-api-access-ztxrv\") pod \"auth-test-model-93079c-kserve-775c79c645-55zgf\" (UID: \"d7a01fa7-260e-466c-b416-aad24c892577\") " pod="ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf"
Oct 30 01:09:40.063848 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:40.062010    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf"
Oct 30 01:09:40.433073 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:40.433041    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf"]
Oct 30 01:09:40.433253 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:09:40.433123    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7a01fa7_260e_466c_b416_aad24c892577.slice/crio-82559695e39d4a80eca4d19d9ea11d76d3d0bf8ea97bc3a7e7fcca91b8424be3 WatchSource:0}: Error finding container 82559695e39d4a80eca4d19d9ea11d76d3d0bf8ea97bc3a7e7fcca91b8424be3: Status 404 returned error can't find the container with id 82559695e39d4a80eca4d19d9ea11d76d3d0bf8ea97bc3a7e7fcca91b8424be3
Oct 30 01:09:40.856054 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:40.855960    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf" event={"ID":"d7a01fa7-260e-466c-b416-aad24c892577","Type":"ContainerStarted","Data":"45fd3fd7fd48861b334246daf47e6084e429dd675eb5c3f8ecc5466a4e3259e8"}
Oct 30 01:09:40.856054 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:40.856010    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf" event={"ID":"d7a01fa7-260e-466c-b416-aad24c892577","Type":"ContainerStarted","Data":"82559695e39d4a80eca4d19d9ea11d76d3d0bf8ea97bc3a7e7fcca91b8424be3"}
Oct 30 01:09:41.063502 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:41.063458    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf"
Oct 30 01:09:47.621669 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:47.621614    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf" podStartSLOduration=8.621600965 podStartE2EDuration="8.621600965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:40.91969168 +0000 UTC m=+2271.307368087" watchObservedRunningTime="2026-10-30 01:09:47.621600965 +0000 UTC m=+2278.009277353"
Oct 30 01:09:47.626430 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:47.626404    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb"]
Oct 30 01:09:47.631270 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:47.631250    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb"
Oct 30 01:09:47.654252 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:47.654221    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb"]
Oct 30 01:09:47.658667 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:47.658254    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-73f667-serving-cert\""
Oct 30 01:09:47.732615 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:47.732576    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c3ad599-c376-47ab-bbf0-fe6c859d6c9c-metrics-certs\") pod \"maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb\" (UID: \"2c3ad599-c376-47ab-bbf0-fe6c859d6c9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb"
Oct 30 01:09:47.732905 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:47.732880    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gk2j\" (UniqueName: \"kubernetes.io/projected/2c3ad599-c376-47ab-bbf0-fe6c859d6c9c-kube-api-access-9gk2j\") pod \"maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb\" (UID: \"2c3ad599-c376-47ab-bbf0-fe6c859d6c9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb"
Oct 30 01:09:47.733065 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:47.733050    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c3ad599-c376-47ab-bbf0-fe6c859d6c9c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb\" (UID: \"2c3ad599-c376-47ab-bbf0-fe6c859d6c9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb"
Oct 30 01:09:47.733146 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:47.733100    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c3ad599-c376-47ab-bbf0-fe6c859d6c9c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb\" (UID: \"2c3ad599-c376-47ab-bbf0-fe6c859d6c9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb"
Oct 30 01:09:47.858082 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:47.840045    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gk2j\" (UniqueName: \"kubernetes.io/projected/2c3ad599-c376-47ab-bbf0-fe6c859d6c9c-kube-api-access-9gk2j\") pod \"maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb\" (UID: \"2c3ad599-c376-47ab-bbf0-fe6c859d6c9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb"
Oct 30 01:09:47.858082 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:47.856299    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c3ad599-c376-47ab-bbf0-fe6c859d6c9c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb\" (UID: \"2c3ad599-c376-47ab-bbf0-fe6c859d6c9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb"
Oct 30 01:09:47.858082 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:47.856372    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c3ad599-c376-47ab-bbf0-fe6c859d6c9c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb\" (UID: \"2c3ad599-c376-47ab-bbf0-fe6c859d6c9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb"
Oct 30 01:09:47.858082 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:47.856460    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c3ad599-c376-47ab-bbf0-fe6c859d6c9c-metrics-certs\") pod \"maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb\" (UID: \"2c3ad599-c376-47ab-bbf0-fe6c859d6c9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb"
Oct 30 01:09:47.858082 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:47.857857    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c3ad599-c376-47ab-bbf0-fe6c859d6c9c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb\" (UID: \"2c3ad599-c376-47ab-bbf0-fe6c859d6c9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb"
Oct 30 01:09:47.858082 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:09:47.857973    4393 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f667-serving-cert: secret "maas-api-e2e-ipp-a-73f667-serving-cert" not found
Oct 30 01:09:47.858082 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:09:47.858046    4393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c3ad599-c376-47ab-bbf0-fe6c859d6c9c-maas-api-tls podName:2c3ad599-c376-47ab-bbf0-fe6c859d6c9c nodeName:}" failed. No retries permitted until 2026-10-30 01:09:48.358026942 +0000 UTC m=+2278.745703327 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2c3ad599-c376-47ab-bbf0-fe6c859d6c9c-maas-api-tls") pod "maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb" (UID: "2c3ad599-c376-47ab-bbf0-fe6c859d6c9c") : secret "maas-api-e2e-ipp-a-73f667-serving-cert" not found
Oct 30 01:09:47.896058 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:47.880999    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gk2j\" (UniqueName: \"kubernetes.io/projected/2c3ad599-c376-47ab-bbf0-fe6c859d6c9c-kube-api-access-9gk2j\") pod \"maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb\" (UID: \"2c3ad599-c376-47ab-bbf0-fe6c859d6c9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb"
Oct 30 01:09:47.897312 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:47.897279    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c3ad599-c376-47ab-bbf0-fe6c859d6c9c-metrics-certs\") pod \"maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb\" (UID: \"2c3ad599-c376-47ab-bbf0-fe6c859d6c9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb"
Oct 30 01:09:48.364054 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:48.363970    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c3ad599-c376-47ab-bbf0-fe6c859d6c9c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb\" (UID: \"2c3ad599-c376-47ab-bbf0-fe6c859d6c9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb"
Oct 30 01:09:48.366659 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:48.366631    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c3ad599-c376-47ab-bbf0-fe6c859d6c9c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb\" (UID: \"2c3ad599-c376-47ab-bbf0-fe6c859d6c9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb"
Oct 30 01:09:48.547493 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:48.547457    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb"
Oct 30 01:09:48.925263 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:48.925231    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb"]
Oct 30 01:09:48.932662 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:09:48.929916    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c3ad599_c376_47ab_bbf0_fe6c859d6c9c.slice/crio-97205c0bc0d2cdedf5102c0c3a20de3775c785e106ecd48c26c88b3144722446 WatchSource:0}: Error finding container 97205c0bc0d2cdedf5102c0c3a20de3775c785e106ecd48c26c88b3144722446: Status 404 returned error can't find the container with id 97205c0bc0d2cdedf5102c0c3a20de3775c785e106ecd48c26c88b3144722446
Oct 30 01:09:48.955209 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:48.955165    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb" event={"ID":"2c3ad599-c376-47ab-bbf0-fe6c859d6c9c","Type":"ContainerStarted","Data":"97205c0bc0d2cdedf5102c0c3a20de3775c785e106ecd48c26c88b3144722446"}
Oct 30 01:09:49.965358 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:49.965318    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb" event={"ID":"2c3ad599-c376-47ab-bbf0-fe6c859d6c9c","Type":"ContainerStarted","Data":"0f31035d565898cfe4448ce2da149ab4d1959d4c1ba6916bbfccff7a98f021c3"}
Oct 30 01:09:49.965761 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:49.965386    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb"
Oct 30 01:09:50.009304 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:50.009238    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb" podStartSLOduration=2.463515548 podStartE2EDuration="3.009217498s" totalImagesPullingTime="545.70195ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:50.007170231 +0000 UTC m=+2280.394846621" watchObservedRunningTime="2026-10-30 01:09:50.009217498 +0000 UTC m=+2280.396893888"
Oct 30 01:09:50.063389 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:50.063357    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf"
Oct 30 01:09:50.075936 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:50.075909    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf"
Oct 30 01:09:50.991109 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:50.991080    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf"
Oct 30 01:09:53.967700 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:53.967661    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"]
Oct 30 01:09:53.968258 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:53.967871    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9" podUID="7afd2398-4294-4236-95d4-82b86a99ca43" containerName="istio-proxy" containerID="cri-o://e0e6169ef6d8353b19049d09b7d5b9e43eadd484e6010553454711b2f32ec8f5" gracePeriod=30
Oct 30 01:09:54.996772 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:54.996735    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"]
Oct 30 01:09:54.997325 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:54.996961    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp" podUID="b6806a61-7f2d-410f-adab-680fdc4a9b08" containerName="istio-proxy" containerID="cri-o://de79eb65c4f6bc042c4cb5938ab390e46be7dfe47151123d9948e3239c2ca8e0" gracePeriod=30
Oct 30 01:09:55.984822 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:55.981468    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb"
Oct 30 01:09:56.515699 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:09:56.515664    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:09:57.015153 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:57.015112    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht"]
Oct 30 01:09:57.020047 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:57.020026    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht"
Oct 30 01:09:57.041673 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:57.041646    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht"]
Oct 30 01:09:57.058112 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:57.058071    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-f64dc0-serving-cert\""
Oct 30 01:09:57.136295 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:57.136259    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4014c968-b2d9-4189-b8fb-18de72da909b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht\" (UID: \"4014c968-b2d9-4189-b8fb-18de72da909b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht"
Oct 30 01:09:57.136475 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:57.136443    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4014c968-b2d9-4189-b8fb-18de72da909b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht\" (UID: \"4014c968-b2d9-4189-b8fb-18de72da909b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht"
Oct 30 01:09:57.136589 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:57.136571    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4014c968-b2d9-4189-b8fb-18de72da909b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht\" (UID: \"4014c968-b2d9-4189-b8fb-18de72da909b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht"
Oct 30 01:09:57.136640 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:57.136627    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh59h\" (UniqueName: \"kubernetes.io/projected/4014c968-b2d9-4189-b8fb-18de72da909b-kube-api-access-qh59h\") pod \"maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht\" (UID: \"4014c968-b2d9-4189-b8fb-18de72da909b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht"
Oct 30 01:09:57.239036 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:57.238989    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4014c968-b2d9-4189-b8fb-18de72da909b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht\" (UID: \"4014c968-b2d9-4189-b8fb-18de72da909b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht"
Oct 30 01:09:57.242951 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:57.242872    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4014c968-b2d9-4189-b8fb-18de72da909b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht\" (UID: \"4014c968-b2d9-4189-b8fb-18de72da909b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht"
Oct 30 01:09:57.244079 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:57.244056    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh59h\" (UniqueName: \"kubernetes.io/projected/4014c968-b2d9-4189-b8fb-18de72da909b-kube-api-access-qh59h\") pod \"maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht\" (UID: \"4014c968-b2d9-4189-b8fb-18de72da909b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht"
Oct 30 01:09:57.244644 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:57.244620    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4014c968-b2d9-4189-b8fb-18de72da909b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht\" (UID: \"4014c968-b2d9-4189-b8fb-18de72da909b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht"
Oct 30 01:09:57.246322 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:57.243936    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4014c968-b2d9-4189-b8fb-18de72da909b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht\" (UID: \"4014c968-b2d9-4189-b8fb-18de72da909b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht"
Oct 30 01:09:57.246597 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:09:57.239709    4393 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f64dc0-serving-cert: secret "maas-api-e2e-shared-a-w2-f64dc0-serving-cert" not found
Oct 30 01:09:57.255999 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:09:57.255975    4393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4014c968-b2d9-4189-b8fb-18de72da909b-maas-api-tls podName:4014c968-b2d9-4189-b8fb-18de72da909b nodeName:}" failed. No retries permitted until 2026-10-30 01:09:57.755953238 +0000 UTC m=+2288.143629618 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4014c968-b2d9-4189-b8fb-18de72da909b-maas-api-tls") pod "maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht" (UID: "4014c968-b2d9-4189-b8fb-18de72da909b") : secret "maas-api-e2e-shared-a-w2-f64dc0-serving-cert" not found
Oct 30 01:09:57.256557 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:57.255849    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4014c968-b2d9-4189-b8fb-18de72da909b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht\" (UID: \"4014c968-b2d9-4189-b8fb-18de72da909b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht"
Oct 30 01:09:57.277668 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:57.276033    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh59h\" (UniqueName: \"kubernetes.io/projected/4014c968-b2d9-4189-b8fb-18de72da909b-kube-api-access-qh59h\") pod \"maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht\" (UID: \"4014c968-b2d9-4189-b8fb-18de72da909b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht"
Oct 30 01:09:57.853918 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:57.853882    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4014c968-b2d9-4189-b8fb-18de72da909b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht\" (UID: \"4014c968-b2d9-4189-b8fb-18de72da909b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht"
Oct 30 01:09:57.856610 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:57.856578    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4014c968-b2d9-4189-b8fb-18de72da909b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht\" (UID: \"4014c968-b2d9-4189-b8fb-18de72da909b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht"
Oct 30 01:09:57.938051 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:57.938014    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht"
Oct 30 01:09:58.247387 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:09:58.247358    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4014c968_b2d9_4189_b8fb_18de72da909b.slice/crio-600dfa94853eb0a68e2e005d7d602fdcf15a65b313a58c8bbec05edce1c33b1e WatchSource:0}: Error finding container 600dfa94853eb0a68e2e005d7d602fdcf15a65b313a58c8bbec05edce1c33b1e: Status 404 returned error can't find the container with id 600dfa94853eb0a68e2e005d7d602fdcf15a65b313a58c8bbec05edce1c33b1e
Oct 30 01:09:58.248060 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:58.248034    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht"]
Oct 30 01:09:59.061006 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:59.060904    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht" event={"ID":"4014c968-b2d9-4189-b8fb-18de72da909b","Type":"ContainerStarted","Data":"e7f0a1b8006c4e898a13a45b2cd110b6d3c6f975a2fd72281ec9b86da3058362"}
Oct 30 01:09:59.061006 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:59.060980    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht" event={"ID":"4014c968-b2d9-4189-b8fb-18de72da909b","Type":"ContainerStarted","Data":"600dfa94853eb0a68e2e005d7d602fdcf15a65b313a58c8bbec05edce1c33b1e"}
Oct 30 01:09:59.061006 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:59.061000    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht"
Oct 30 01:09:59.063150 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:59.063121    4393 generic.go:356] "Generic (PLEG): container finished" podID="7afd2398-4294-4236-95d4-82b86a99ca43" containerID="e0e6169ef6d8353b19049d09b7d5b9e43eadd484e6010553454711b2f32ec8f5" exitCode=0
Oct 30 01:09:59.063270 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:59.063154    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9" event={"ID":"7afd2398-4294-4236-95d4-82b86a99ca43","Type":"ContainerDied","Data":"e0e6169ef6d8353b19049d09b7d5b9e43eadd484e6010553454711b2f32ec8f5"}
Oct 30 01:09:59.101955 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:59.101873    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht" podStartSLOduration=2.720106091 podStartE2EDuration="3.101859277s" totalImagesPullingTime="381.753186ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:09:59.10046384 +0000 UTC m=+2289.488140230" watchObservedRunningTime="2026-10-30 01:09:59.101859277 +0000 UTC m=+2289.489535667"
Oct 30 01:09:59.257306 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:59.256447    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"
Oct 30 01:09:59.302622 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:59.302579    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7afd2398-4294-4236-95d4-82b86a99ca43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7afd2398-4294-4236-95d4-82b86a99ca43-istio-podinfo\") pod \"7afd2398-4294-4236-95d4-82b86a99ca43\" (UID: \"7afd2398-4294-4236-95d4-82b86a99ca43\") "
Oct 30 01:09:59.302845 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:59.302691    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7afd2398-4294-4236-95d4-82b86a99ca43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7afd2398-4294-4236-95d4-82b86a99ca43-istio-data\") pod \"7afd2398-4294-4236-95d4-82b86a99ca43\" (UID: \"7afd2398-4294-4236-95d4-82b86a99ca43\") "
Oct 30 01:09:59.302845 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:59.302728    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7afd2398-4294-4236-95d4-82b86a99ca43-kube-api-access-h5z7v\" (UniqueName: \"kubernetes.io/projected/7afd2398-4294-4236-95d4-82b86a99ca43-kube-api-access-h5z7v\") pod \"7afd2398-4294-4236-95d4-82b86a99ca43\" (UID: \"7afd2398-4294-4236-95d4-82b86a99ca43\") "
Oct 30 01:09:59.302845 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:59.302761    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7afd2398-4294-4236-95d4-82b86a99ca43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7afd2398-4294-4236-95d4-82b86a99ca43-istiod-ca-cert\") pod \"7afd2398-4294-4236-95d4-82b86a99ca43\" (UID: \"7afd2398-4294-4236-95d4-82b86a99ca43\") "
Oct 30 01:09:59.303051 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:59.302860    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7afd2398-4294-4236-95d4-82b86a99ca43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7afd2398-4294-4236-95d4-82b86a99ca43-istio-envoy\") pod \"7afd2398-4294-4236-95d4-82b86a99ca43\" (UID: \"7afd2398-4294-4236-95d4-82b86a99ca43\") "
Oct 30 01:09:59.303051 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:59.302931    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7afd2398-4294-4236-95d4-82b86a99ca43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7afd2398-4294-4236-95d4-82b86a99ca43-credential-socket\") pod \"7afd2398-4294-4236-95d4-82b86a99ca43\" (UID: \"7afd2398-4294-4236-95d4-82b86a99ca43\") "
Oct 30 01:09:59.303051 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:59.302959    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7afd2398-4294-4236-95d4-82b86a99ca43-istio-data" pod "7afd2398-4294-4236-95d4-82b86a99ca43" (UID: "7afd2398-4294-4236-95d4-82b86a99ca43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:59.303051 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:59.303040    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7afd2398-4294-4236-95d4-82b86a99ca43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7afd2398-4294-4236-95d4-82b86a99ca43-workload-certs\") pod \"7afd2398-4294-4236-95d4-82b86a99ca43\" (UID: \"7afd2398-4294-4236-95d4-82b86a99ca43\") "
Oct 30 01:09:59.303272 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:59.303079    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7afd2398-4294-4236-95d4-82b86a99ca43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7afd2398-4294-4236-95d4-82b86a99ca43-workload-socket\") pod \"7afd2398-4294-4236-95d4-82b86a99ca43\" (UID: \"7afd2398-4294-4236-95d4-82b86a99ca43\") "
Oct 30 01:09:59.303272 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:59.303112    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7afd2398-4294-4236-95d4-82b86a99ca43-istio-token\" (UniqueName: \"kubernetes.io/projected/7afd2398-4294-4236-95d4-82b86a99ca43-istio-token\") pod \"7afd2398-4294-4236-95d4-82b86a99ca43\" (UID: \"7afd2398-4294-4236-95d4-82b86a99ca43\") "
Oct 30 01:09:59.304418 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:59.303773    4393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7afd2398-4294-4236-95d4-82b86a99ca43-istio-data\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:09:59.304418 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:59.304115    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7afd2398-4294-4236-95d4-82b86a99ca43-credential-socket" pod "7afd2398-4294-4236-95d4-82b86a99ca43" (UID: "7afd2398-4294-4236-95d4-82b86a99ca43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:59.304418 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:59.304188    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7afd2398-4294-4236-95d4-82b86a99ca43-workload-socket" pod "7afd2398-4294-4236-95d4-82b86a99ca43" (UID: "7afd2398-4294-4236-95d4-82b86a99ca43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:59.304643 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:59.304557    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7afd2398-4294-4236-95d4-82b86a99ca43-workload-certs" pod "7afd2398-4294-4236-95d4-82b86a99ca43" (UID: "7afd2398-4294-4236-95d4-82b86a99ca43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:59.305729 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:59.305697    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7afd2398-4294-4236-95d4-82b86a99ca43-istiod-ca-cert" pod "7afd2398-4294-4236-95d4-82b86a99ca43" (UID: "7afd2398-4294-4236-95d4-82b86a99ca43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:09:59.306361 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:59.306311    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7afd2398-4294-4236-95d4-82b86a99ca43-istio-token" pod "7afd2398-4294-4236-95d4-82b86a99ca43" (UID: "7afd2398-4294-4236-95d4-82b86a99ca43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:59.307058 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:59.307036    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7afd2398-4294-4236-95d4-82b86a99ca43-kube-api-access-h5z7v" pod "7afd2398-4294-4236-95d4-82b86a99ca43" (UID: "7afd2398-4294-4236-95d4-82b86a99ca43"). InnerVolumeSpecName "kube-api-access-h5z7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:09:59.307715 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:59.307691    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7afd2398-4294-4236-95d4-82b86a99ca43-istio-podinfo" pod "7afd2398-4294-4236-95d4-82b86a99ca43" (UID: "7afd2398-4294-4236-95d4-82b86a99ca43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:09:59.310544 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:59.310524    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7afd2398-4294-4236-95d4-82b86a99ca43-istio-envoy" pod "7afd2398-4294-4236-95d4-82b86a99ca43" (UID: "7afd2398-4294-4236-95d4-82b86a99ca43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:09:59.405283 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:59.405252    4393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7afd2398-4294-4236-95d4-82b86a99ca43-credential-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:09:59.405283 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:59.405281    4393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7afd2398-4294-4236-95d4-82b86a99ca43-workload-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:09:59.405283 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:59.405291    4393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7afd2398-4294-4236-95d4-82b86a99ca43-workload-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:09:59.405518 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:59.405299    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7afd2398-4294-4236-95d4-82b86a99ca43-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:09:59.405518 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:59.405309    4393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7afd2398-4294-4236-95d4-82b86a99ca43-istio-podinfo\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:09:59.405518 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:59.405324    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5z7v\" (UniqueName: \"kubernetes.io/projected/7afd2398-4294-4236-95d4-82b86a99ca43-kube-api-access-h5z7v\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:09:59.405518 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:59.405334    4393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7afd2398-4294-4236-95d4-82b86a99ca43-istiod-ca-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:09:59.405518 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:59.405342    4393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7afd2398-4294-4236-95d4-82b86a99ca43-istio-envoy\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:09:59.536986 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:59.536906    4393 patch_prober.go:29] interesting pod/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:09:59.536986 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:09:59.536961    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp" podUID="b6806a61-7f2d-410f-adab-680fdc4a9b08" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:10:00.074262 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:00.074234    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"
Oct 30 01:10:00.074730 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:00.074256    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9" event={"ID":"7afd2398-4294-4236-95d4-82b86a99ca43","Type":"ContainerDied","Data":"ad0f2229a433d598d4833444c97a0080708f723e3fb2587f1269af74d16eb580"}
Oct 30 01:10:00.074730 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:00.074323    4393 scope.go:122] "RemoveContainer" containerID="e0e6169ef6d8353b19049d09b7d5b9e43eadd484e6010553454711b2f32ec8f5"
Oct 30 01:10:00.163723 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:00.163683    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"]
Oct 30 01:10:00.171843 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:00.171813    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6f4fba-openshift-default-65f44b9749-gwlz9"]
Oct 30 01:10:00.309947 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:00.309917    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"
Oct 30 01:10:00.324017 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:00.323975    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6806a61-7f2d-410f-adab-680fdc4a9b08-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6806a61-7f2d-410f-adab-680fdc4a9b08-workload-socket\") pod \"b6806a61-7f2d-410f-adab-680fdc4a9b08\" (UID: \"b6806a61-7f2d-410f-adab-680fdc4a9b08\") "
Oct 30 01:10:00.324183 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:00.324034    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6806a61-7f2d-410f-adab-680fdc4a9b08-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6806a61-7f2d-410f-adab-680fdc4a9b08-istiod-ca-cert\") pod \"b6806a61-7f2d-410f-adab-680fdc4a9b08\" (UID: \"b6806a61-7f2d-410f-adab-680fdc4a9b08\") "
Oct 30 01:10:00.324183 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:00.324134    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6806a61-7f2d-410f-adab-680fdc4a9b08-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6806a61-7f2d-410f-adab-680fdc4a9b08-istio-envoy\") pod \"b6806a61-7f2d-410f-adab-680fdc4a9b08\" (UID: \"b6806a61-7f2d-410f-adab-680fdc4a9b08\") "
Oct 30 01:10:00.324326 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:00.324205    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6806a61-7f2d-410f-adab-680fdc4a9b08-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6806a61-7f2d-410f-adab-680fdc4a9b08-istio-data\") pod \"b6806a61-7f2d-410f-adab-680fdc4a9b08\" (UID: \"b6806a61-7f2d-410f-adab-680fdc4a9b08\") "
Oct 30 01:10:00.324326 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:00.324243    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6806a61-7f2d-410f-adab-680fdc4a9b08-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6806a61-7f2d-410f-adab-680fdc4a9b08-credential-socket\") pod \"b6806a61-7f2d-410f-adab-680fdc4a9b08\" (UID: \"b6806a61-7f2d-410f-adab-680fdc4a9b08\") "
Oct 30 01:10:00.324326 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:00.324251    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6806a61-7f2d-410f-adab-680fdc4a9b08-workload-socket" pod "b6806a61-7f2d-410f-adab-680fdc4a9b08" (UID: "b6806a61-7f2d-410f-adab-680fdc4a9b08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:00.324326 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:00.324307    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6806a61-7f2d-410f-adab-680fdc4a9b08-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6806a61-7f2d-410f-adab-680fdc4a9b08-workload-certs\") pod \"b6806a61-7f2d-410f-adab-680fdc4a9b08\" (UID: \"b6806a61-7f2d-410f-adab-680fdc4a9b08\") "
Oct 30 01:10:00.324523 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:00.324435    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6806a61-7f2d-410f-adab-680fdc4a9b08-istio-token\" (UniqueName: \"kubernetes.io/projected/b6806a61-7f2d-410f-adab-680fdc4a9b08-istio-token\") pod \"b6806a61-7f2d-410f-adab-680fdc4a9b08\" (UID: \"b6806a61-7f2d-410f-adab-680fdc4a9b08\") "
Oct 30 01:10:00.324523 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:00.324482    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6806a61-7f2d-410f-adab-680fdc4a9b08-istio-data" pod "b6806a61-7f2d-410f-adab-680fdc4a9b08" (UID: "b6806a61-7f2d-410f-adab-680fdc4a9b08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:00.324523 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:00.324499    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6806a61-7f2d-410f-adab-680fdc4a9b08-istiod-ca-cert" pod "b6806a61-7f2d-410f-adab-680fdc4a9b08" (UID: "b6806a61-7f2d-410f-adab-680fdc4a9b08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:10:00.324737 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:00.324565    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6806a61-7f2d-410f-adab-680fdc4a9b08-kube-api-access-wjglw\" (UniqueName: \"kubernetes.io/projected/b6806a61-7f2d-410f-adab-680fdc4a9b08-kube-api-access-wjglw\") pod \"b6806a61-7f2d-410f-adab-680fdc4a9b08\" (UID: \"b6806a61-7f2d-410f-adab-680fdc4a9b08\") "
Oct 30 01:10:00.324737 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:00.324716    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6806a61-7f2d-410f-adab-680fdc4a9b08-credential-socket" pod "b6806a61-7f2d-410f-adab-680fdc4a9b08" (UID: "b6806a61-7f2d-410f-adab-680fdc4a9b08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:00.324884 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:00.324731    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6806a61-7f2d-410f-adab-680fdc4a9b08-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6806a61-7f2d-410f-adab-680fdc4a9b08-istio-podinfo\") pod \"b6806a61-7f2d-410f-adab-680fdc4a9b08\" (UID: \"b6806a61-7f2d-410f-adab-680fdc4a9b08\") "
Oct 30 01:10:00.324958 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:00.324902    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6806a61-7f2d-410f-adab-680fdc4a9b08-workload-certs" pod "b6806a61-7f2d-410f-adab-680fdc4a9b08" (UID: "b6806a61-7f2d-410f-adab-680fdc4a9b08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:00.325486 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:00.325461    4393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6806a61-7f2d-410f-adab-680fdc4a9b08-istio-data\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:10:00.325486 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:00.325492    4393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6806a61-7f2d-410f-adab-680fdc4a9b08-credential-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:10:00.325652 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:00.325510    4393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6806a61-7f2d-410f-adab-680fdc4a9b08-workload-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:10:00.325725 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:00.325633    4393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6806a61-7f2d-410f-adab-680fdc4a9b08-workload-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:10:00.325725 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:00.325714    4393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6806a61-7f2d-410f-adab-680fdc4a9b08-istiod-ca-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:10:00.326714 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:00.326686    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6806a61-7f2d-410f-adab-680fdc4a9b08-istio-envoy" pod "b6806a61-7f2d-410f-adab-680fdc4a9b08" (UID: "b6806a61-7f2d-410f-adab-680fdc4a9b08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:00.326932 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:00.326912    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6806a61-7f2d-410f-adab-680fdc4a9b08-istio-token" pod "b6806a61-7f2d-410f-adab-680fdc4a9b08" (UID: "b6806a61-7f2d-410f-adab-680fdc4a9b08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:00.327013 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:00.326964    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6806a61-7f2d-410f-adab-680fdc4a9b08-kube-api-access-wjglw" pod "b6806a61-7f2d-410f-adab-680fdc4a9b08" (UID: "b6806a61-7f2d-410f-adab-680fdc4a9b08"). InnerVolumeSpecName "kube-api-access-wjglw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:00.327236 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:00.327215    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6806a61-7f2d-410f-adab-680fdc4a9b08-istio-podinfo" pod "b6806a61-7f2d-410f-adab-680fdc4a9b08" (UID: "b6806a61-7f2d-410f-adab-680fdc4a9b08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:10:00.427456 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:00.427418    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6806a61-7f2d-410f-adab-680fdc4a9b08-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:10:00.427456 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:00.427453    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjglw\" (UniqueName: \"kubernetes.io/projected/b6806a61-7f2d-410f-adab-680fdc4a9b08-kube-api-access-wjglw\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:10:00.427658 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:00.427465    4393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6806a61-7f2d-410f-adab-680fdc4a9b08-istio-podinfo\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:10:00.427658 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:00.427484    4393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6806a61-7f2d-410f-adab-680fdc4a9b08-istio-envoy\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:10:01.086011 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:01.085921    4393 generic.go:356] "Generic (PLEG): container finished" podID="b6806a61-7f2d-410f-adab-680fdc4a9b08" containerID="de79eb65c4f6bc042c4cb5938ab390e46be7dfe47151123d9948e3239c2ca8e0" exitCode=0
Oct 30 01:10:01.086011 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:01.085983    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"
Oct 30 01:10:01.086011 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:01.085993    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp" event={"ID":"b6806a61-7f2d-410f-adab-680fdc4a9b08","Type":"ContainerDied","Data":"de79eb65c4f6bc042c4cb5938ab390e46be7dfe47151123d9948e3239c2ca8e0"}
Oct 30 01:10:01.086640 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:01.086070    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp" event={"ID":"b6806a61-7f2d-410f-adab-680fdc4a9b08","Type":"ContainerDied","Data":"c6446b96b94565729edf7f098c4128da4551005fad01c022e08c98c67fffe7eb"}
Oct 30 01:10:01.086640 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:01.086106    4393 scope.go:122] "RemoveContainer" containerID="de79eb65c4f6bc042c4cb5938ab390e46be7dfe47151123d9948e3239c2ca8e0"
Oct 30 01:10:01.103504 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:01.103481    4393 scope.go:122] "RemoveContainer" containerID="de79eb65c4f6bc042c4cb5938ab390e46be7dfe47151123d9948e3239c2ca8e0"
Oct 30 01:10:01.103854 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:10:01.103833    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de79eb65c4f6bc042c4cb5938ab390e46be7dfe47151123d9948e3239c2ca8e0\": container with ID starting with de79eb65c4f6bc042c4cb5938ab390e46be7dfe47151123d9948e3239c2ca8e0 not found: ID does not exist" containerID="de79eb65c4f6bc042c4cb5938ab390e46be7dfe47151123d9948e3239c2ca8e0"
Oct 30 01:10:01.103922 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:01.103862    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de79eb65c4f6bc042c4cb5938ab390e46be7dfe47151123d9948e3239c2ca8e0"} err="failed to get container status \"de79eb65c4f6bc042c4cb5938ab390e46be7dfe47151123d9948e3239c2ca8e0\": rpc error: code = NotFound desc = could not find container \"de79eb65c4f6bc042c4cb5938ab390e46be7dfe47151123d9948e3239c2ca8e0\": container with ID starting with de79eb65c4f6bc042c4cb5938ab390e46be7dfe47151123d9948e3239c2ca8e0 not found: ID does not exist"
Oct 30 01:10:01.139371 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:01.139338    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"]
Oct 30 01:10:01.154218 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:01.154184    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-hsssp"]
Oct 30 01:10:01.727890 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:01.727851    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7afd2398-4294-4236-95d4-82b86a99ca43" path="/var/lib/kubelet/pods/7afd2398-4294-4236-95d4-82b86a99ca43/volumes"
Oct 30 01:10:01.728559 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:01.728534    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6806a61-7f2d-410f-adab-680fdc4a9b08" path="/var/lib/kubelet/pods/b6806a61-7f2d-410f-adab-680fdc4a9b08/volumes"
Oct 30 01:10:05.080703 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:05.080674    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht"
Oct 30 01:10:07.549877 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.549837    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"]
Oct 30 01:10:07.550766 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.550734    4393 cpu_manager.go:414] "containerMap: removing container" podUID="7afd2398-4294-4236-95d4-82b86a99ca43" containerName="istio-proxy"
Oct 30 01:10:07.550766 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.550761    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7afd2398-4294-4236-95d4-82b86a99ca43" containerName="istio-proxy"
Oct 30 01:10:07.550969 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.550823    4393 cpu_manager.go:414] "containerMap: removing container" podUID="b6806a61-7f2d-410f-adab-680fdc4a9b08" containerName="istio-proxy"
Oct 30 01:10:07.550969 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.550829    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6806a61-7f2d-410f-adab-680fdc4a9b08" containerName="istio-proxy"
Oct 30 01:10:07.550969 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.550963    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6806a61-7f2d-410f-adab-680fdc4a9b08" containerName="istio-proxy"
Oct 30 01:10:07.551097 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.551004    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="7afd2398-4294-4236-95d4-82b86a99ca43" containerName="istio-proxy"
Oct 30 01:10:07.556618 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.556599    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"
Oct 30 01:10:07.582424 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.582386    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-11f521-openshift-default-dockercfg-vkflh\""
Oct 30 01:10:07.589886 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.589856    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"]
Oct 30 01:10:07.639708 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.639672    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b31f25ea-0447-4caf-9237-d59594938f72-istiod-ca-cert\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286\" (UID: \"b31f25ea-0447-4caf-9237-d59594938f72\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"
Oct 30 01:10:07.639881 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.639737    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b31f25ea-0447-4caf-9237-d59594938f72-istio-podinfo\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286\" (UID: \"b31f25ea-0447-4caf-9237-d59594938f72\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"
Oct 30 01:10:07.639961 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.639939    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31f25ea-0447-4caf-9237-d59594938f72-credential-socket\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286\" (UID: \"b31f25ea-0447-4caf-9237-d59594938f72\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"
Oct 30 01:10:07.639995 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.639974    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b31f25ea-0447-4caf-9237-d59594938f72-istio-envoy\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286\" (UID: \"b31f25ea-0447-4caf-9237-d59594938f72\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"
Oct 30 01:10:07.640059 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.640048    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b31f25ea-0447-4caf-9237-d59594938f72-istio-token\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286\" (UID: \"b31f25ea-0447-4caf-9237-d59594938f72\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"
Oct 30 01:10:07.640141 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.640122    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31f25ea-0447-4caf-9237-d59594938f72-workload-socket\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286\" (UID: \"b31f25ea-0447-4caf-9237-d59594938f72\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"
Oct 30 01:10:07.640232 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.640219    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq5n8\" (UniqueName: \"kubernetes.io/projected/b31f25ea-0447-4caf-9237-d59594938f72-kube-api-access-wq5n8\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286\" (UID: \"b31f25ea-0447-4caf-9237-d59594938f72\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"
Oct 30 01:10:07.640269 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.640250    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b31f25ea-0447-4caf-9237-d59594938f72-istio-data\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286\" (UID: \"b31f25ea-0447-4caf-9237-d59594938f72\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"
Oct 30 01:10:07.640331 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.640318    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b31f25ea-0447-4caf-9237-d59594938f72-workload-certs\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286\" (UID: \"b31f25ea-0447-4caf-9237-d59594938f72\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"
Oct 30 01:10:07.741926 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.741892    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b31f25ea-0447-4caf-9237-d59594938f72-istio-data\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286\" (UID: \"b31f25ea-0447-4caf-9237-d59594938f72\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"
Oct 30 01:10:07.742101 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.741948    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b31f25ea-0447-4caf-9237-d59594938f72-workload-certs\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286\" (UID: \"b31f25ea-0447-4caf-9237-d59594938f72\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"
Oct 30 01:10:07.742101 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.741968    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b31f25ea-0447-4caf-9237-d59594938f72-istiod-ca-cert\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286\" (UID: \"b31f25ea-0447-4caf-9237-d59594938f72\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"
Oct 30 01:10:07.742101 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.742009    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b31f25ea-0447-4caf-9237-d59594938f72-istio-podinfo\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286\" (UID: \"b31f25ea-0447-4caf-9237-d59594938f72\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"
Oct 30 01:10:07.742101 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.742090    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31f25ea-0447-4caf-9237-d59594938f72-credential-socket\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286\" (UID: \"b31f25ea-0447-4caf-9237-d59594938f72\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"
Oct 30 01:10:07.742342 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.742115    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b31f25ea-0447-4caf-9237-d59594938f72-istio-envoy\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286\" (UID: \"b31f25ea-0447-4caf-9237-d59594938f72\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"
Oct 30 01:10:07.742342 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.742168    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b31f25ea-0447-4caf-9237-d59594938f72-istio-token\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286\" (UID: \"b31f25ea-0447-4caf-9237-d59594938f72\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"
Oct 30 01:10:07.742342 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.742228    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31f25ea-0447-4caf-9237-d59594938f72-workload-socket\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286\" (UID: \"b31f25ea-0447-4caf-9237-d59594938f72\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"
Oct 30 01:10:07.742342 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.742326    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq5n8\" (UniqueName: \"kubernetes.io/projected/b31f25ea-0447-4caf-9237-d59594938f72-kube-api-access-wq5n8\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286\" (UID: \"b31f25ea-0447-4caf-9237-d59594938f72\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"
Oct 30 01:10:07.742518 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.742354    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b31f25ea-0447-4caf-9237-d59594938f72-istio-data\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286\" (UID: \"b31f25ea-0447-4caf-9237-d59594938f72\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"
Oct 30 01:10:07.742518 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.742399    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b31f25ea-0447-4caf-9237-d59594938f72-workload-certs\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286\" (UID: \"b31f25ea-0447-4caf-9237-d59594938f72\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"
Oct 30 01:10:07.742518 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.742462    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31f25ea-0447-4caf-9237-d59594938f72-credential-socket\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286\" (UID: \"b31f25ea-0447-4caf-9237-d59594938f72\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"
Oct 30 01:10:07.742746 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.742724    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31f25ea-0447-4caf-9237-d59594938f72-workload-socket\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286\" (UID: \"b31f25ea-0447-4caf-9237-d59594938f72\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"
Oct 30 01:10:07.742861 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.742841    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b31f25ea-0447-4caf-9237-d59594938f72-istiod-ca-cert\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286\" (UID: \"b31f25ea-0447-4caf-9237-d59594938f72\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"
Oct 30 01:10:07.744724 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.744693    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b31f25ea-0447-4caf-9237-d59594938f72-istio-envoy\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286\" (UID: \"b31f25ea-0447-4caf-9237-d59594938f72\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"
Oct 30 01:10:07.744889 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.744817    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b31f25ea-0447-4caf-9237-d59594938f72-istio-podinfo\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286\" (UID: \"b31f25ea-0447-4caf-9237-d59594938f72\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"
Oct 30 01:10:07.765415 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.765381    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b31f25ea-0447-4caf-9237-d59594938f72-istio-token\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286\" (UID: \"b31f25ea-0447-4caf-9237-d59594938f72\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"
Oct 30 01:10:07.765599 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.765575    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq5n8\" (UniqueName: \"kubernetes.io/projected/b31f25ea-0447-4caf-9237-d59594938f72-kube-api-access-wq5n8\") pod \"e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286\" (UID: \"b31f25ea-0447-4caf-9237-d59594938f72\") " pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"
Oct 30 01:10:07.871061 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:07.871030    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"
Oct 30 01:10:08.200558 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:10:08.200530    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb31f25ea_0447_4caf_9237_d59594938f72.slice/crio-62b676676f1e1679464503da34171bb3a705581a60d48b4dbfb1060fd90899a2 WatchSource:0}: Error finding container 62b676676f1e1679464503da34171bb3a705581a60d48b4dbfb1060fd90899a2: Status 404 returned error can't find the container with id 62b676676f1e1679464503da34171bb3a705581a60d48b4dbfb1060fd90899a2
Oct 30 01:10:08.201612 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:08.201519    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"]
Oct 30 01:10:08.203901 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:08.203871    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:10:08.203990 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:08.203936    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:10:08.203990 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:08.203965    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:10:09.170046 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:09.170009    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286" event={"ID":"b31f25ea-0447-4caf-9237-d59594938f72","Type":"ContainerStarted","Data":"11cdd267f301af8315ba6fd9a2f4d97f011f70a7ad8f546fc15b2727d20ae911"}
Oct 30 01:10:09.170480 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:09.170074    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286" event={"ID":"b31f25ea-0447-4caf-9237-d59594938f72","Type":"ContainerStarted","Data":"62b676676f1e1679464503da34171bb3a705581a60d48b4dbfb1060fd90899a2"}
Oct 30 01:10:09.239431 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:09.239377    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286" podStartSLOduration=2.23936148 podStartE2EDuration="2.23936148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:10:09.223901339 +0000 UTC m=+2299.611577729" watchObservedRunningTime="2026-10-30 01:10:09.23936148 +0000 UTC m=+2299.627037868"
Oct 30 01:10:09.839305 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:09.839270    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj"]
Oct 30 01:10:09.871957 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:09.871921    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"
Oct 30 01:10:09.873335 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:09.873307    4393 patch_prober.go:29] interesting pod/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 30 01:10:09.873454 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:09.873357    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286" podUID="b31f25ea-0447-4caf-9237-d59594938f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 30 01:10:10.871916 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:10.871883    4393 patch_prober.go:29] interesting pod/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 30 01:10:10.872426 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:10.871944    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286" podUID="b31f25ea-0447-4caf-9237-d59594938f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 30 01:10:10.999151 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:10.999118    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz"]
Oct 30 01:10:11.871556 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:11.871521    4393 patch_prober.go:29] interesting pod/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 30 01:10:11.871737 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:11.871576    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286" podUID="b31f25ea-0447-4caf-9237-d59594938f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 30 01:10:12.871720 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:12.871682    4393 patch_prober.go:29] interesting pod/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 30 01:10:12.872151 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:12.871733    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286" podUID="b31f25ea-0447-4caf-9237-d59594938f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 30 01:10:13.871952 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:13.871661    4393 patch_prober.go:29] interesting pod/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 30 01:10:13.871952 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:13.871729    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286" podUID="b31f25ea-0447-4caf-9237-d59594938f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 30 01:10:14.871543 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:14.871506    4393 patch_prober.go:29] interesting pod/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Oct 30 01:10:14.871716 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:14.871569    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286" podUID="b31f25ea-0447-4caf-9237-d59594938f72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Oct 30 01:10:15.876792 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:15.876739    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"
Oct 30 01:10:15.877276 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:15.877262    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"
Oct 30 01:10:15.877963 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:15.877937    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"
Oct 30 01:10:24.864008 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:24.863968    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj" podUID="e5c1b663-2684-4d99-bf83-75ff9c347e99" containerName="main" containerID="cri-o://aaf16a755305154e3a5b0e88aa6040f66bd7d027d244ca98342906d227f922f1" gracePeriod=45
Oct 30 01:10:25.149438 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:25.149409    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj"
Oct 30 01:10:25.276284 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:25.276253    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c1b663-2684-4d99-bf83-75ff9c347e99-home\" (UniqueName: \"kubernetes.io/empty-dir/e5c1b663-2684-4d99-bf83-75ff9c347e99-home\") pod \"e5c1b663-2684-4d99-bf83-75ff9c347e99\" (UID: \"e5c1b663-2684-4d99-bf83-75ff9c347e99\") "
Oct 30 01:10:25.276490 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:25.276293    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c1b663-2684-4d99-bf83-75ff9c347e99-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5c1b663-2684-4d99-bf83-75ff9c347e99-dshm\") pod \"e5c1b663-2684-4d99-bf83-75ff9c347e99\" (UID: \"e5c1b663-2684-4d99-bf83-75ff9c347e99\") "
Oct 30 01:10:25.276490 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:25.276314    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5c1b663-2684-4d99-bf83-75ff9c347e99-tls-certs\" (UniqueName: \"kubernetes.io/secret/e5c1b663-2684-4d99-bf83-75ff9c347e99-tls-certs\") pod \"e5c1b663-2684-4d99-bf83-75ff9c347e99\" (UID: \"e5c1b663-2684-4d99-bf83-75ff9c347e99\") "
Oct 30 01:10:25.276490 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:25.276371    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c1b663-2684-4d99-bf83-75ff9c347e99-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5c1b663-2684-4d99-bf83-75ff9c347e99-tmp-dir\") pod \"e5c1b663-2684-4d99-bf83-75ff9c347e99\" (UID: \"e5c1b663-2684-4d99-bf83-75ff9c347e99\") "
Oct 30 01:10:25.276490 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:25.276461    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5c1b663-2684-4d99-bf83-75ff9c347e99-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5c1b663-2684-4d99-bf83-75ff9c347e99-model-cache\") pod \"e5c1b663-2684-4d99-bf83-75ff9c347e99\" (UID: \"e5c1b663-2684-4d99-bf83-75ff9c347e99\") "
Oct 30 01:10:25.276723 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:25.276534    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5c1b663-2684-4d99-bf83-75ff9c347e99-kube-api-access-g4npz\" (UniqueName: \"kubernetes.io/projected/e5c1b663-2684-4d99-bf83-75ff9c347e99-kube-api-access-g4npz\") pod \"e5c1b663-2684-4d99-bf83-75ff9c347e99\" (UID: \"e5c1b663-2684-4d99-bf83-75ff9c347e99\") "
Oct 30 01:10:25.276723 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:25.276551    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c1b663-2684-4d99-bf83-75ff9c347e99-home" pod "e5c1b663-2684-4d99-bf83-75ff9c347e99" (UID: "e5c1b663-2684-4d99-bf83-75ff9c347e99"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:25.276832 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:25.276712    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c1b663-2684-4d99-bf83-75ff9c347e99-tmp-dir" pod "e5c1b663-2684-4d99-bf83-75ff9c347e99" (UID: "e5c1b663-2684-4d99-bf83-75ff9c347e99"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:25.276832 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:25.276774    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c1b663-2684-4d99-bf83-75ff9c347e99-model-cache" pod "e5c1b663-2684-4d99-bf83-75ff9c347e99" (UID: "e5c1b663-2684-4d99-bf83-75ff9c347e99"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:25.277321 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:25.277302    4393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5c1b663-2684-4d99-bf83-75ff9c347e99-home\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:10:25.277392 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:25.277323    4393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5c1b663-2684-4d99-bf83-75ff9c347e99-tmp-dir\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:10:25.277392 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:25.277334    4393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5c1b663-2684-4d99-bf83-75ff9c347e99-model-cache\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:10:25.279057 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:25.279031    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c1b663-2684-4d99-bf83-75ff9c347e99-kube-api-access-g4npz" pod "e5c1b663-2684-4d99-bf83-75ff9c347e99" (UID: "e5c1b663-2684-4d99-bf83-75ff9c347e99"). InnerVolumeSpecName "kube-api-access-g4npz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:25.279166 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:25.279147    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c1b663-2684-4d99-bf83-75ff9c347e99-dshm" pod "e5c1b663-2684-4d99-bf83-75ff9c347e99" (UID: "e5c1b663-2684-4d99-bf83-75ff9c347e99"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:25.279166 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:25.279155    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5c1b663-2684-4d99-bf83-75ff9c347e99-tls-certs" pod "e5c1b663-2684-4d99-bf83-75ff9c347e99" (UID: "e5c1b663-2684-4d99-bf83-75ff9c347e99"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:10:25.379597 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:25.379504    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4npz\" (UniqueName: \"kubernetes.io/projected/e5c1b663-2684-4d99-bf83-75ff9c347e99-kube-api-access-g4npz\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:10:25.379597 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:25.379542    4393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5c1b663-2684-4d99-bf83-75ff9c347e99-dshm\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:10:25.379597 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:25.379556    4393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5c1b663-2684-4d99-bf83-75ff9c347e99-tls-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:10:25.381433 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:25.381397    4393 generic.go:356] "Generic (PLEG): container finished" podID="e5c1b663-2684-4d99-bf83-75ff9c347e99" containerID="aaf16a755305154e3a5b0e88aa6040f66bd7d027d244ca98342906d227f922f1" exitCode=0
Oct 30 01:10:25.381584 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:25.381440    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj" event={"ID":"e5c1b663-2684-4d99-bf83-75ff9c347e99","Type":"ContainerDied","Data":"aaf16a755305154e3a5b0e88aa6040f66bd7d027d244ca98342906d227f922f1"}
Oct 30 01:10:25.381584 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:25.381471    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj" event={"ID":"e5c1b663-2684-4d99-bf83-75ff9c347e99","Type":"ContainerDied","Data":"7619d4971ac06d51739c064c128093f91da1a2865b56a777acb345df517ab13a"}
Oct 30 01:10:25.381584 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:25.381446    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj"
Oct 30 01:10:25.381584 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:25.381494    4393 scope.go:122] "RemoveContainer" containerID="aaf16a755305154e3a5b0e88aa6040f66bd7d027d244ca98342906d227f922f1"
Oct 30 01:10:25.394962 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:25.394935    4393 scope.go:122] "RemoveContainer" containerID="aaf16a755305154e3a5b0e88aa6040f66bd7d027d244ca98342906d227f922f1"
Oct 30 01:10:25.395296 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:10:25.395271    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaf16a755305154e3a5b0e88aa6040f66bd7d027d244ca98342906d227f922f1\": container with ID starting with aaf16a755305154e3a5b0e88aa6040f66bd7d027d244ca98342906d227f922f1 not found: ID does not exist" containerID="aaf16a755305154e3a5b0e88aa6040f66bd7d027d244ca98342906d227f922f1"
Oct 30 01:10:25.395357 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:25.395308    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaf16a755305154e3a5b0e88aa6040f66bd7d027d244ca98342906d227f922f1"} err="failed to get container status \"aaf16a755305154e3a5b0e88aa6040f66bd7d027d244ca98342906d227f922f1\": rpc error: code = NotFound desc = could not find container \"aaf16a755305154e3a5b0e88aa6040f66bd7d027d244ca98342906d227f922f1\": container with ID starting with aaf16a755305154e3a5b0e88aa6040f66bd7d027d244ca98342906d227f922f1 not found: ID does not exist"
Oct 30 01:10:25.433488 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:25.433453    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj"]
Oct 30 01:10:25.438457 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:25.438428    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-beeba4/auth-test-model-beeba4-kserve-65f6dcfb89-cqxmj"]
Oct 30 01:10:25.776722 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:25.776633    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5c1b663-2684-4d99-bf83-75ff9c347e99" path="/var/lib/kubelet/pods/e5c1b663-2684-4d99-bf83-75ff9c347e99/volumes"
Oct 30 01:10:26.049893 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:26.049804    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz" podUID="bc530394-4e4b-4d7f-b031-65139f579756" containerName="main" containerID="cri-o://1885f5b74627a6ef2f702a1f1c0ba3a37598f6b4d37859c4e9b8cfb2d98bc520" gracePeriod=45
Oct 30 01:10:26.189467 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:26.189430    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz" podUID="bc530394-4e4b-4d7f-b031-65139f579756" containerName="main" probeResult="failure" output="Get \"https://10.133.0.162:8000/ready\": dial tcp 10.133.0.162:8000: connect: connection refused"
Oct 30 01:10:26.345393 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:26.345368    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz"
Oct 30 01:10:26.394504 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:26.394464    4393 generic.go:356] "Generic (PLEG): container finished" podID="bc530394-4e4b-4d7f-b031-65139f579756" containerID="1885f5b74627a6ef2f702a1f1c0ba3a37598f6b4d37859c4e9b8cfb2d98bc520" exitCode=0
Oct 30 01:10:26.394711 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:26.394519    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz"
Oct 30 01:10:26.394711 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:26.394544    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz" event={"ID":"bc530394-4e4b-4d7f-b031-65139f579756","Type":"ContainerDied","Data":"1885f5b74627a6ef2f702a1f1c0ba3a37598f6b4d37859c4e9b8cfb2d98bc520"}
Oct 30 01:10:26.394711 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:26.394600    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz" event={"ID":"bc530394-4e4b-4d7f-b031-65139f579756","Type":"ContainerDied","Data":"11ed37d329ce730fdf8692bb4a9e20a4f1d855d12bf2d53dfce85d68ff450e6e"}
Oct 30 01:10:26.394711 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:26.394618    4393 scope.go:122] "RemoveContainer" containerID="1885f5b74627a6ef2f702a1f1c0ba3a37598f6b4d37859c4e9b8cfb2d98bc520"
Oct 30 01:10:26.409971 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:26.409947    4393 scope.go:122] "RemoveContainer" containerID="1885f5b74627a6ef2f702a1f1c0ba3a37598f6b4d37859c4e9b8cfb2d98bc520"
Oct 30 01:10:26.410398 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:10:26.410370    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1885f5b74627a6ef2f702a1f1c0ba3a37598f6b4d37859c4e9b8cfb2d98bc520\": container with ID starting with 1885f5b74627a6ef2f702a1f1c0ba3a37598f6b4d37859c4e9b8cfb2d98bc520 not found: ID does not exist" containerID="1885f5b74627a6ef2f702a1f1c0ba3a37598f6b4d37859c4e9b8cfb2d98bc520"
Oct 30 01:10:26.410471 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:26.410411    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1885f5b74627a6ef2f702a1f1c0ba3a37598f6b4d37859c4e9b8cfb2d98bc520"} err="failed to get container status \"1885f5b74627a6ef2f702a1f1c0ba3a37598f6b4d37859c4e9b8cfb2d98bc520\": rpc error: code = NotFound desc = could not find container \"1885f5b74627a6ef2f702a1f1c0ba3a37598f6b4d37859c4e9b8cfb2d98bc520\": container with ID starting with 1885f5b74627a6ef2f702a1f1c0ba3a37598f6b4d37859c4e9b8cfb2d98bc520 not found: ID does not exist"
Oct 30 01:10:26.494896 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:26.494863    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc530394-4e4b-4d7f-b031-65139f579756-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc530394-4e4b-4d7f-b031-65139f579756-model-cache\") pod \"bc530394-4e4b-4d7f-b031-65139f579756\" (UID: \"bc530394-4e4b-4d7f-b031-65139f579756\") "
Oct 30 01:10:26.495105 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:26.495081    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc530394-4e4b-4d7f-b031-65139f579756-tls-certs\" (UniqueName: \"kubernetes.io/secret/bc530394-4e4b-4d7f-b031-65139f579756-tls-certs\") pod \"bc530394-4e4b-4d7f-b031-65139f579756\" (UID: \"bc530394-4e4b-4d7f-b031-65139f579756\") "
Oct 30 01:10:26.495157 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:26.495118    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc530394-4e4b-4d7f-b031-65139f579756-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc530394-4e4b-4d7f-b031-65139f579756-dshm\") pod \"bc530394-4e4b-4d7f-b031-65139f579756\" (UID: \"bc530394-4e4b-4d7f-b031-65139f579756\") "
Oct 30 01:10:26.495200 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:26.495151    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc530394-4e4b-4d7f-b031-65139f579756-model-cache" pod "bc530394-4e4b-4d7f-b031-65139f579756" (UID: "bc530394-4e4b-4d7f-b031-65139f579756"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:26.495200 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:26.495174    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc530394-4e4b-4d7f-b031-65139f579756-kube-api-access-9chhr\" (UniqueName: \"kubernetes.io/projected/bc530394-4e4b-4d7f-b031-65139f579756-kube-api-access-9chhr\") pod \"bc530394-4e4b-4d7f-b031-65139f579756\" (UID: \"bc530394-4e4b-4d7f-b031-65139f579756\") "
Oct 30 01:10:26.495413 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:26.495243    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc530394-4e4b-4d7f-b031-65139f579756-home\" (UniqueName: \"kubernetes.io/empty-dir/bc530394-4e4b-4d7f-b031-65139f579756-home\") pod \"bc530394-4e4b-4d7f-b031-65139f579756\" (UID: \"bc530394-4e4b-4d7f-b031-65139f579756\") "
Oct 30 01:10:26.495413 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:26.495273    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc530394-4e4b-4d7f-b031-65139f579756-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc530394-4e4b-4d7f-b031-65139f579756-tmp-dir\") pod \"bc530394-4e4b-4d7f-b031-65139f579756\" (UID: \"bc530394-4e4b-4d7f-b031-65139f579756\") "
Oct 30 01:10:26.495697 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:26.495644    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc530394-4e4b-4d7f-b031-65139f579756-tmp-dir" pod "bc530394-4e4b-4d7f-b031-65139f579756" (UID: "bc530394-4e4b-4d7f-b031-65139f579756"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:26.496360 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:26.496327    4393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc530394-4e4b-4d7f-b031-65139f579756-tmp-dir\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:10:26.496483 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:26.496361    4393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc530394-4e4b-4d7f-b031-65139f579756-model-cache\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:10:26.496483 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:26.496378    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc530394-4e4b-4d7f-b031-65139f579756-home" pod "bc530394-4e4b-4d7f-b031-65139f579756" (UID: "bc530394-4e4b-4d7f-b031-65139f579756"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:26.497767 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:26.497740    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc530394-4e4b-4d7f-b031-65139f579756-tls-certs" pod "bc530394-4e4b-4d7f-b031-65139f579756" (UID: "bc530394-4e4b-4d7f-b031-65139f579756"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:10:26.497899 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:26.497748    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc530394-4e4b-4d7f-b031-65139f579756-dshm" pod "bc530394-4e4b-4d7f-b031-65139f579756" (UID: "bc530394-4e4b-4d7f-b031-65139f579756"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:26.497899 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:26.497870    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc530394-4e4b-4d7f-b031-65139f579756-kube-api-access-9chhr" pod "bc530394-4e4b-4d7f-b031-65139f579756" (UID: "bc530394-4e4b-4d7f-b031-65139f579756"). InnerVolumeSpecName "kube-api-access-9chhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:26.515619 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:10:26.515583    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:10:26.597866 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:26.597737    4393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc530394-4e4b-4d7f-b031-65139f579756-tls-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:10:26.597866 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:26.597769    4393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc530394-4e4b-4d7f-b031-65139f579756-dshm\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:10:26.597866 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:26.597818    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9chhr\" (UniqueName: \"kubernetes.io/projected/bc530394-4e4b-4d7f-b031-65139f579756-kube-api-access-9chhr\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:10:26.597866 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:26.597826    4393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc530394-4e4b-4d7f-b031-65139f579756-home\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:10:26.745804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:26.745738    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz"]
Oct 30 01:10:26.750771 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:26.750741    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-73435a/auth-test-model-73435a-kserve-6cf8b665-2frgz"]
Oct 30 01:10:26.781387 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:10:26.781356    4393 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc530394_4e4b_4d7f_b031_65139f579756.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 01:10:27.727393 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:27.727358    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc530394-4e4b-4d7f-b031-65139f579756" path="/var/lib/kubelet/pods/bc530394-4e4b-4d7f-b031-65139f579756/volumes"
Oct 30 01:10:37.370317 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:37.370276    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7"]
Oct 30 01:10:48.497927 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:48.497885    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn"]
Oct 30 01:10:51.270182 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:51.270152    4393 scope.go:122] "RemoveContainer" containerID="2530e15d554970bf633648513bbe32e408b1ced3ce4390b170b8593257b80e51"
Oct 30 01:10:52.394162 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:52.394121    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7" podUID="8bef7646-518d-4735-bee6-b1f804d5ea8e" containerName="main" containerID="cri-o://ebd8c14f2c520f2d44edaf3de2ce15b1407917cd1ab75c80bff55083c6783576" gracePeriod=45
Oct 30 01:10:52.443353 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:52.443308    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7" podUID="8bef7646-518d-4735-bee6-b1f804d5ea8e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.155:8000/ready\": dial tcp 10.133.0.155:8000: connect: connection refused"
Oct 30 01:10:52.704265 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:52.703896    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7"
Oct 30 01:10:52.871186 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:52.871149    4393 generic.go:356] "Generic (PLEG): container finished" podID="8bef7646-518d-4735-bee6-b1f804d5ea8e" containerID="ebd8c14f2c520f2d44edaf3de2ce15b1407917cd1ab75c80bff55083c6783576" exitCode=0
Oct 30 01:10:52.871520 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:52.871499    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7" event={"ID":"8bef7646-518d-4735-bee6-b1f804d5ea8e","Type":"ContainerDied","Data":"ebd8c14f2c520f2d44edaf3de2ce15b1407917cd1ab75c80bff55083c6783576"}
Oct 30 01:10:52.871633 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:52.871621    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7" event={"ID":"8bef7646-518d-4735-bee6-b1f804d5ea8e","Type":"ContainerDied","Data":"708ffcf160ae8efc9c1838fe969ac974504664ccd420f399c6e455bf3fbbc7f8"}
Oct 30 01:10:52.871732 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:52.871720    4393 scope.go:122] "RemoveContainer" containerID="ebd8c14f2c520f2d44edaf3de2ce15b1407917cd1ab75c80bff55083c6783576"
Oct 30 01:10:52.872009 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:52.871993    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7"
Oct 30 01:10:52.874400 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:52.874367    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bef7646-518d-4735-bee6-b1f804d5ea8e-kube-api-access-9f7l6\" (UniqueName: \"kubernetes.io/projected/8bef7646-518d-4735-bee6-b1f804d5ea8e-kube-api-access-9f7l6\") pod \"8bef7646-518d-4735-bee6-b1f804d5ea8e\" (UID: \"8bef7646-518d-4735-bee6-b1f804d5ea8e\") "
Oct 30 01:10:52.875256 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:52.875214    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bef7646-518d-4735-bee6-b1f804d5ea8e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bef7646-518d-4735-bee6-b1f804d5ea8e-dshm\") pod \"8bef7646-518d-4735-bee6-b1f804d5ea8e\" (UID: \"8bef7646-518d-4735-bee6-b1f804d5ea8e\") "
Oct 30 01:10:52.875982 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:52.875961    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bef7646-518d-4735-bee6-b1f804d5ea8e-home\" (UniqueName: \"kubernetes.io/empty-dir/8bef7646-518d-4735-bee6-b1f804d5ea8e-home\") pod \"8bef7646-518d-4735-bee6-b1f804d5ea8e\" (UID: \"8bef7646-518d-4735-bee6-b1f804d5ea8e\") "
Oct 30 01:10:52.876591 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:52.876570    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bef7646-518d-4735-bee6-b1f804d5ea8e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bef7646-518d-4735-bee6-b1f804d5ea8e-model-cache\") pod \"8bef7646-518d-4735-bee6-b1f804d5ea8e\" (UID: \"8bef7646-518d-4735-bee6-b1f804d5ea8e\") "
Oct 30 01:10:52.877118 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:52.876431    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bef7646-518d-4735-bee6-b1f804d5ea8e-home" pod "8bef7646-518d-4735-bee6-b1f804d5ea8e" (UID: "8bef7646-518d-4735-bee6-b1f804d5ea8e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:52.877355 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:52.876989    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bef7646-518d-4735-bee6-b1f804d5ea8e-model-cache" pod "8bef7646-518d-4735-bee6-b1f804d5ea8e" (UID: "8bef7646-518d-4735-bee6-b1f804d5ea8e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:52.877578 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:52.877557    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bef7646-518d-4735-bee6-b1f804d5ea8e-tls-certs\" (UniqueName: \"kubernetes.io/secret/8bef7646-518d-4735-bee6-b1f804d5ea8e-tls-certs\") pod \"8bef7646-518d-4735-bee6-b1f804d5ea8e\" (UID: \"8bef7646-518d-4735-bee6-b1f804d5ea8e\") "
Oct 30 01:10:52.877911 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:52.877894    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bef7646-518d-4735-bee6-b1f804d5ea8e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bef7646-518d-4735-bee6-b1f804d5ea8e-tmp-dir\") pod \"8bef7646-518d-4735-bee6-b1f804d5ea8e\" (UID: \"8bef7646-518d-4735-bee6-b1f804d5ea8e\") "
Oct 30 01:10:52.878625 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:52.878346    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bef7646-518d-4735-bee6-b1f804d5ea8e-tmp-dir" pod "8bef7646-518d-4735-bee6-b1f804d5ea8e" (UID: "8bef7646-518d-4735-bee6-b1f804d5ea8e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:52.880435 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:52.880416    4393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bef7646-518d-4735-bee6-b1f804d5ea8e-tmp-dir\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:10:52.880885 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:52.880871    4393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bef7646-518d-4735-bee6-b1f804d5ea8e-home\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:10:52.880987 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:52.880976    4393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bef7646-518d-4735-bee6-b1f804d5ea8e-model-cache\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:10:52.924316 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:52.921123    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bef7646-518d-4735-bee6-b1f804d5ea8e-kube-api-access-9f7l6" pod "8bef7646-518d-4735-bee6-b1f804d5ea8e" (UID: "8bef7646-518d-4735-bee6-b1f804d5ea8e"). InnerVolumeSpecName "kube-api-access-9f7l6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:10:52.928927 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:52.928884    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bef7646-518d-4735-bee6-b1f804d5ea8e-tls-certs" pod "8bef7646-518d-4735-bee6-b1f804d5ea8e" (UID: "8bef7646-518d-4735-bee6-b1f804d5ea8e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:10:52.930736 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:52.930708    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bef7646-518d-4735-bee6-b1f804d5ea8e-dshm" pod "8bef7646-518d-4735-bee6-b1f804d5ea8e" (UID: "8bef7646-518d-4735-bee6-b1f804d5ea8e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:10:52.938793 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:52.938709    4393 scope.go:122] "RemoveContainer" containerID="ebd8c14f2c520f2d44edaf3de2ce15b1407917cd1ab75c80bff55083c6783576"
Oct 30 01:10:52.940810 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:10:52.939527    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebd8c14f2c520f2d44edaf3de2ce15b1407917cd1ab75c80bff55083c6783576\": container with ID starting with ebd8c14f2c520f2d44edaf3de2ce15b1407917cd1ab75c80bff55083c6783576 not found: ID does not exist" containerID="ebd8c14f2c520f2d44edaf3de2ce15b1407917cd1ab75c80bff55083c6783576"
Oct 30 01:10:52.940810 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:52.939579    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebd8c14f2c520f2d44edaf3de2ce15b1407917cd1ab75c80bff55083c6783576"} err="failed to get container status \"ebd8c14f2c520f2d44edaf3de2ce15b1407917cd1ab75c80bff55083c6783576\": rpc error: code = NotFound desc = could not find container \"ebd8c14f2c520f2d44edaf3de2ce15b1407917cd1ab75c80bff55083c6783576\": container with ID starting with ebd8c14f2c520f2d44edaf3de2ce15b1407917cd1ab75c80bff55083c6783576 not found: ID does not exist"
Oct 30 01:10:52.982250 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:52.982223    4393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bef7646-518d-4735-bee6-b1f804d5ea8e-tls-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:10:52.982250 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:52.982249    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f7l6\" (UniqueName: \"kubernetes.io/projected/8bef7646-518d-4735-bee6-b1f804d5ea8e-kube-api-access-9f7l6\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:10:52.982415 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:52.982259    4393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bef7646-518d-4735-bee6-b1f804d5ea8e-dshm\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:10:53.221516 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:53.221484    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7"]
Oct 30 01:10:53.232265 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:53.232236    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-11f521/sub-test-model-11f521-kserve-57cd86885c-nkgt7"]
Oct 30 01:10:53.729138 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:10:53.729102    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bef7646-518d-4735-bee6-b1f804d5ea8e" path="/var/lib/kubelet/pods/8bef7646-518d-4735-bee6-b1f804d5ea8e/volumes"
Oct 30 01:10:56.519173 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:10:56.519129    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:11:03.521907 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:03.521860    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn" podUID="ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2" containerName="main" containerID="cri-o://85a40f9a35d83c9ba1ac70f65eda02937a982365859998f9514f7c3bf02730be" gracePeriod=45
Oct 30 01:11:03.817553 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:03.817524    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn"
Oct 30 01:11:03.875752 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:03.875715    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-tls-certs\" (UniqueName: \"kubernetes.io/secret/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-tls-certs\") pod \"ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2\" (UID: \"ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2\") "
Oct 30 01:11:03.875946 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:03.875889    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-tmp-dir\") pod \"ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2\" (UID: \"ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2\") "
Oct 30 01:11:03.876027 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:03.876010    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-kube-api-access-r8wfd\" (UniqueName: \"kubernetes.io/projected/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-kube-api-access-r8wfd\") pod \"ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2\" (UID: \"ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2\") "
Oct 30 01:11:03.876080 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:03.876052    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-home\" (UniqueName: \"kubernetes.io/empty-dir/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-home\") pod \"ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2\" (UID: \"ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2\") "
Oct 30 01:11:03.876134 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:03.876087    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-model-cache\") pod \"ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2\" (UID: \"ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2\") "
Oct 30 01:11:03.876199 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:03.876168    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-dshm\") pod \"ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2\" (UID: \"ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2\") "
Oct 30 01:11:03.876253 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:03.876190    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-tmp-dir" pod "ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2" (UID: "ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:11:03.876620 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:03.876302    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-home" pod "ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2" (UID: "ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:11:03.876620 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:03.876412    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-model-cache" pod "ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2" (UID: "ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:11:03.876957 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:03.876934    4393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-home\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:11:03.877138 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:03.877115    4393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-model-cache\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:11:03.877234 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:03.877144    4393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-tmp-dir\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:11:03.878819 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:03.878758    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-kube-api-access-r8wfd" pod "ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2" (UID: "ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2"). InnerVolumeSpecName "kube-api-access-r8wfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:11:03.878939 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:03.878859    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-tls-certs" pod "ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2" (UID: "ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:11:03.879896 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:03.879870    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-dshm" pod "ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2" (UID: "ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:11:03.979201 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:03.979167    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8wfd\" (UniqueName: \"kubernetes.io/projected/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-kube-api-access-r8wfd\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:11:03.979201 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:03.979202    4393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-dshm\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:11:03.979390 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:03.979217    4393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2-tls-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:11:04.032238 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:04.032158    4393 generic.go:356] "Generic (PLEG): container finished" podID="ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2" containerID="85a40f9a35d83c9ba1ac70f65eda02937a982365859998f9514f7c3bf02730be" exitCode=0
Oct 30 01:11:04.032238 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:04.032206    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn"
Oct 30 01:11:04.032433 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:04.032247    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn" event={"ID":"ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2","Type":"ContainerDied","Data":"85a40f9a35d83c9ba1ac70f65eda02937a982365859998f9514f7c3bf02730be"}
Oct 30 01:11:04.032433 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:04.032289    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn" event={"ID":"ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2","Type":"ContainerDied","Data":"8a516e765a04bf936a0defab518856b5608f98361d6ea7e91361b313bc7cf7b9"}
Oct 30 01:11:04.032433 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:04.032311    4393 scope.go:122] "RemoveContainer" containerID="85a40f9a35d83c9ba1ac70f65eda02937a982365859998f9514f7c3bf02730be"
Oct 30 01:11:04.044082 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:04.044064    4393 scope.go:122] "RemoveContainer" containerID="85a40f9a35d83c9ba1ac70f65eda02937a982365859998f9514f7c3bf02730be"
Oct 30 01:11:04.044338 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:11:04.044321    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85a40f9a35d83c9ba1ac70f65eda02937a982365859998f9514f7c3bf02730be\": container with ID starting with 85a40f9a35d83c9ba1ac70f65eda02937a982365859998f9514f7c3bf02730be not found: ID does not exist" containerID="85a40f9a35d83c9ba1ac70f65eda02937a982365859998f9514f7c3bf02730be"
Oct 30 01:11:04.044407 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:04.044345    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85a40f9a35d83c9ba1ac70f65eda02937a982365859998f9514f7c3bf02730be"} err="failed to get container status \"85a40f9a35d83c9ba1ac70f65eda02937a982365859998f9514f7c3bf02730be\": rpc error: code = NotFound desc = could not find container \"85a40f9a35d83c9ba1ac70f65eda02937a982365859998f9514f7c3bf02730be\": container with ID starting with 85a40f9a35d83c9ba1ac70f65eda02937a982365859998f9514f7c3bf02730be not found: ID does not exist"
Oct 30 01:11:04.084435 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:04.084405    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn"]
Oct 30 01:11:04.093630 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:04.091654    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8e0f66/sub-test-model-8e0f66-kserve-74dffc749f-nhfqn"]
Oct 30 01:11:05.726608 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:05.726529    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2" path="/var/lib/kubelet/pods/ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2/volumes"
Oct 30 01:11:13.710804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:13.705882    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-68746cfdb7-hsxp2"]
Oct 30 01:11:13.710804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:13.707568    4393 cpu_manager.go:414] "containerMap: removing container" podUID="ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2" containerName="main"
Oct 30 01:11:13.710804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:13.707593    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2" containerName="main"
Oct 30 01:11:13.710804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:13.707614    4393 cpu_manager.go:414] "containerMap: removing container" podUID="e5c1b663-2684-4d99-bf83-75ff9c347e99" containerName="main"
Oct 30 01:11:13.710804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:13.707622    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5c1b663-2684-4d99-bf83-75ff9c347e99" containerName="main"
Oct 30 01:11:13.710804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:13.707657    4393 cpu_manager.go:414] "containerMap: removing container" podUID="8bef7646-518d-4735-bee6-b1f804d5ea8e" containerName="main"
Oct 30 01:11:13.710804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:13.707666    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bef7646-518d-4735-bee6-b1f804d5ea8e" containerName="main"
Oct 30 01:11:13.710804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:13.707680    4393 cpu_manager.go:414] "containerMap: removing container" podUID="bc530394-4e4b-4d7f-b031-65139f579756" containerName="main"
Oct 30 01:11:13.710804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:13.707687    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc530394-4e4b-4d7f-b031-65139f579756" containerName="main"
Oct 30 01:11:13.710804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:13.707888    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddc80cf1-533f-4b1c-8d51-cdef7ec8f1c2" containerName="main"
Oct 30 01:11:13.710804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:13.707905    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5c1b663-2684-4d99-bf83-75ff9c347e99" containerName="main"
Oct 30 01:11:13.710804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:13.707921    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bef7646-518d-4735-bee6-b1f804d5ea8e" containerName="main"
Oct 30 01:11:13.710804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:13.707929    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc530394-4e4b-4d7f-b031-65139f579756" containerName="main"
Oct 30 01:11:13.811848 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:13.803284    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-68746cfdb7-hsxp2"]
Oct 30 01:11:13.811848 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:13.808966    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-68746cfdb7-hsxp2"
Oct 30 01:11:13.839691 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:13.839451    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-73f667\""
Oct 30 01:11:13.839691 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:13.839564    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-73f667-dockercfg-pk7h5\""
Oct 30 01:11:13.997185 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:13.997096    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-73f667-68746cfdb7-hsxp2\" (UID: \"8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-68746cfdb7-hsxp2"
Oct 30 01:11:13.997373 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:13.997291    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h4sf\" (UniqueName: \"kubernetes.io/projected/8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a-kube-api-access-4h4sf\") pod \"payload-pre-processing-e2e-ipp-a-73f667-68746cfdb7-hsxp2\" (UID: \"8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-68746cfdb7-hsxp2"
Oct 30 01:11:14.099140 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:14.099102    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-73f667-68746cfdb7-hsxp2\" (UID: \"8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-68746cfdb7-hsxp2"
Oct 30 01:11:14.099340 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:14.099256    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h4sf\" (UniqueName: \"kubernetes.io/projected/8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a-kube-api-access-4h4sf\") pod \"payload-pre-processing-e2e-ipp-a-73f667-68746cfdb7-hsxp2\" (UID: \"8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-68746cfdb7-hsxp2"
Oct 30 01:11:14.099934 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:14.099911    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-73f667-68746cfdb7-hsxp2\" (UID: \"8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-68746cfdb7-hsxp2"
Oct 30 01:11:14.122150 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:14.122122    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h4sf\" (UniqueName: \"kubernetes.io/projected/8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a-kube-api-access-4h4sf\") pod \"payload-pre-processing-e2e-ipp-a-73f667-68746cfdb7-hsxp2\" (UID: \"8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-68746cfdb7-hsxp2"
Oct 30 01:11:14.194867 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:14.194831    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-68746cfdb7-hsxp2"
Oct 30 01:11:14.504076 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:11:14.504015    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d81b7b6_1c1f_43dc_82c9_1ccc444a7b5a.slice/crio-e8aa273278c405a09ec87104c1100a2fadd94334fd33f37812d1c1f50f73eb1d WatchSource:0}: Error finding container e8aa273278c405a09ec87104c1100a2fadd94334fd33f37812d1c1f50f73eb1d: Status 404 returned error can't find the container with id e8aa273278c405a09ec87104c1100a2fadd94334fd33f37812d1c1f50f73eb1d
Oct 30 01:11:14.504966 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:14.504937    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-68746cfdb7-hsxp2"]
Oct 30 01:11:14.506118 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:14.506100    4393 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 01:11:15.188574 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:15.188536    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-68746cfdb7-hsxp2" event={"ID":"8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a","Type":"ContainerStarted","Data":"e8aa273278c405a09ec87104c1100a2fadd94334fd33f37812d1c1f50f73eb1d"}
Oct 30 01:11:17.207762 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:17.207701    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-68746cfdb7-hsxp2" event={"ID":"8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a","Type":"ContainerStarted","Data":"8e2c4f087ae3925b97cf39a9a161436fda2755954ebc485e501479d19a4f660e"}
Oct 30 01:11:17.208289 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:17.207797    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-68746cfdb7-hsxp2"
Oct 30 01:11:17.252135 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:17.252075    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-68746cfdb7-hsxp2" podStartSLOduration=2.213660542 podStartE2EDuration="4.252058409s" totalImagesPullingTime="2.038397867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:17.248485293 +0000 UTC m=+2367.636161682" watchObservedRunningTime="2026-10-30 01:11:17.252058409 +0000 UTC m=+2367.639734799"
Oct 30 01:11:18.345831 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:18.345773    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8822-4bab5b"]
Oct 30 01:11:18.350913 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:18.350887    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8822-4bab5b"
Oct 30 01:11:18.367982 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:18.367953    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8822-4bab5b"]
Oct 30 01:11:18.372828 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:18.372805    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nrxkf\""
Oct 30 01:11:18.510495 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:18.507615    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk9wr\" (UniqueName: \"kubernetes.io/projected/b65e76c5-7825-4900-9e59-fae76d726f1f-kube-api-access-lk9wr\") pod \"test-curl-8822-4bab5b\" (UID: \"b65e76c5-7825-4900-9e59-fae76d726f1f\") " pod="opendatahub/test-curl-8822-4bab5b"
Oct 30 01:11:18.663016 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:18.662974    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk9wr\" (UniqueName: \"kubernetes.io/projected/b65e76c5-7825-4900-9e59-fae76d726f1f-kube-api-access-lk9wr\") pod \"test-curl-8822-4bab5b\" (UID: \"b65e76c5-7825-4900-9e59-fae76d726f1f\") " pod="opendatahub/test-curl-8822-4bab5b"
Oct 30 01:11:18.687098 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:18.687063    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk9wr\" (UniqueName: \"kubernetes.io/projected/b65e76c5-7825-4900-9e59-fae76d726f1f-kube-api-access-lk9wr\") pod \"test-curl-8822-4bab5b\" (UID: \"b65e76c5-7825-4900-9e59-fae76d726f1f\") " pod="opendatahub/test-curl-8822-4bab5b"
Oct 30 01:11:18.965506 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:18.965413    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8822-4bab5b"
Oct 30 01:11:19.278082 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:11:19.278043    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb65e76c5_7825_4900_9e59_fae76d726f1f.slice/crio-9e3fe0539196b8eb5c2866a40b7bf465be78ba0b30045d1094a10577609f52e0 WatchSource:0}: Error finding container 9e3fe0539196b8eb5c2866a40b7bf465be78ba0b30045d1094a10577609f52e0: Status 404 returned error can't find the container with id 9e3fe0539196b8eb5c2866a40b7bf465be78ba0b30045d1094a10577609f52e0
Oct 30 01:11:19.278973 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:19.278939    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8822-4bab5b"]
Oct 30 01:11:20.244454 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:20.244418    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8822-4bab5b" event={"ID":"b65e76c5-7825-4900-9e59-fae76d726f1f","Type":"ContainerStarted","Data":"d33ce9587964489f2cb846f0d59a860d479e911fa32d6568e17c558e9a3c1d59"}
Oct 30 01:11:20.244454 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:20.244460    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8822-4bab5b" event={"ID":"b65e76c5-7825-4900-9e59-fae76d726f1f","Type":"ContainerStarted","Data":"9e3fe0539196b8eb5c2866a40b7bf465be78ba0b30045d1094a10577609f52e0"}
Oct 30 01:11:20.282484 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:20.282441    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8822-4bab5b" podStartSLOduration=2.282427227 podStartE2EDuration="2.282427227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:20.281359129 +0000 UTC m=+2370.669035517" watchObservedRunningTime="2026-10-30 01:11:20.282427227 +0000 UTC m=+2370.670103616"
Oct 30 01:11:21.243085 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:21.243048    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8822-4bab5b"]
Oct 30 01:11:21.248181 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:21.248148    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8822-4bab5b"]
Oct 30 01:11:22.187932 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:22.187896    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt"]
Oct 30 01:11:22.188964 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:22.188940    4393 cpu_manager.go:414] "containerMap: removing container" podUID="b65e76c5-7825-4900-9e59-fae76d726f1f" containerName="test-curl-8822-4bab5b"
Oct 30 01:11:22.188964 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:22.188961    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b65e76c5-7825-4900-9e59-fae76d726f1f" containerName="test-curl-8822-4bab5b"
Oct 30 01:11:22.189195 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:22.189076    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="b65e76c5-7825-4900-9e59-fae76d726f1f" containerName="test-curl-8822-4bab5b"
Oct 30 01:11:22.193207 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:22.193184    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt"
Oct 30 01:11:22.212600 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:22.212569    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt"]
Oct 30 01:11:22.216884 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:22.216764    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-73435a\"/\"kube-root-ca.crt\""
Oct 30 01:11:22.216884 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:22.216852    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-73435a\"/\"openshift-service-ca.crt\""
Oct 30 01:11:22.218879 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:22.218829    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-73435a\"/\"sub-test-model-73435a-kserve-self-signed-certs\""
Oct 30 01:11:22.219028 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:22.219010    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-73435a\"/\"default-dockercfg-w9cql\""
Oct 30 01:11:22.256226 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:22.256185    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gmj9\" (UniqueName: \"kubernetes.io/projected/1b3371ba-4d68-47a2-b45d-ef2470346e48-kube-api-access-7gmj9\") pod \"sub-test-model-73435a-kserve-57db64f59-vrxlt\" (UID: \"1b3371ba-4d68-47a2-b45d-ef2470346e48\") " pod="ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt"
Oct 30 01:11:22.256678 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:22.256508    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b3371ba-4d68-47a2-b45d-ef2470346e48-tls-certs\") pod \"sub-test-model-73435a-kserve-57db64f59-vrxlt\" (UID: \"1b3371ba-4d68-47a2-b45d-ef2470346e48\") " pod="ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt"
Oct 30 01:11:22.256678 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:22.256571    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b3371ba-4d68-47a2-b45d-ef2470346e48-home\") pod \"sub-test-model-73435a-kserve-57db64f59-vrxlt\" (UID: \"1b3371ba-4d68-47a2-b45d-ef2470346e48\") " pod="ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt"
Oct 30 01:11:22.256855 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:22.256725    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b3371ba-4d68-47a2-b45d-ef2470346e48-dshm\") pod \"sub-test-model-73435a-kserve-57db64f59-vrxlt\" (UID: \"1b3371ba-4d68-47a2-b45d-ef2470346e48\") " pod="ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt"
Oct 30 01:11:22.256855 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:22.256848    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b3371ba-4d68-47a2-b45d-ef2470346e48-tmp-dir\") pod \"sub-test-model-73435a-kserve-57db64f59-vrxlt\" (UID: \"1b3371ba-4d68-47a2-b45d-ef2470346e48\") " pod="ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt"
Oct 30 01:11:22.257023 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:22.257004    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b3371ba-4d68-47a2-b45d-ef2470346e48-model-cache\") pod \"sub-test-model-73435a-kserve-57db64f59-vrxlt\" (UID: \"1b3371ba-4d68-47a2-b45d-ef2470346e48\") " pod="ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt"
Oct 30 01:11:22.263333 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:22.263297    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8822-4bab5b" podUID="b65e76c5-7825-4900-9e59-fae76d726f1f" containerName="test-curl-8822-4bab5b" containerID="cri-o://d33ce9587964489f2cb846f0d59a860d479e911fa32d6568e17c558e9a3c1d59" gracePeriod=2
Oct 30 01:11:22.358131 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:22.358091    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b3371ba-4d68-47a2-b45d-ef2470346e48-tls-certs\") pod \"sub-test-model-73435a-kserve-57db64f59-vrxlt\" (UID: \"1b3371ba-4d68-47a2-b45d-ef2470346e48\") " pod="ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt"
Oct 30 01:11:22.358322 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:22.358141    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b3371ba-4d68-47a2-b45d-ef2470346e48-home\") pod \"sub-test-model-73435a-kserve-57db64f59-vrxlt\" (UID: \"1b3371ba-4d68-47a2-b45d-ef2470346e48\") " pod="ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt"
Oct 30 01:11:22.358322 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:22.358204    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b3371ba-4d68-47a2-b45d-ef2470346e48-dshm\") pod \"sub-test-model-73435a-kserve-57db64f59-vrxlt\" (UID: \"1b3371ba-4d68-47a2-b45d-ef2470346e48\") " pod="ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt"
Oct 30 01:11:22.358322 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:22.358255    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b3371ba-4d68-47a2-b45d-ef2470346e48-tmp-dir\") pod \"sub-test-model-73435a-kserve-57db64f59-vrxlt\" (UID: \"1b3371ba-4d68-47a2-b45d-ef2470346e48\") " pod="ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt"
Oct 30 01:11:22.358486 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:22.358354    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b3371ba-4d68-47a2-b45d-ef2470346e48-model-cache\") pod \"sub-test-model-73435a-kserve-57db64f59-vrxlt\" (UID: \"1b3371ba-4d68-47a2-b45d-ef2470346e48\") " pod="ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt"
Oct 30 01:11:22.358486 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:22.358438    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gmj9\" (UniqueName: \"kubernetes.io/projected/1b3371ba-4d68-47a2-b45d-ef2470346e48-kube-api-access-7gmj9\") pod \"sub-test-model-73435a-kserve-57db64f59-vrxlt\" (UID: \"1b3371ba-4d68-47a2-b45d-ef2470346e48\") " pod="ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt"
Oct 30 01:11:22.358711 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:22.358692    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b3371ba-4d68-47a2-b45d-ef2470346e48-home\") pod \"sub-test-model-73435a-kserve-57db64f59-vrxlt\" (UID: \"1b3371ba-4d68-47a2-b45d-ef2470346e48\") " pod="ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt"
Oct 30 01:11:22.358844 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:22.358726    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b3371ba-4d68-47a2-b45d-ef2470346e48-tmp-dir\") pod \"sub-test-model-73435a-kserve-57db64f59-vrxlt\" (UID: \"1b3371ba-4d68-47a2-b45d-ef2470346e48\") " pod="ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt"
Oct 30 01:11:22.358908 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:22.358849    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b3371ba-4d68-47a2-b45d-ef2470346e48-model-cache\") pod \"sub-test-model-73435a-kserve-57db64f59-vrxlt\" (UID: \"1b3371ba-4d68-47a2-b45d-ef2470346e48\") " pod="ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt"
Oct 30 01:11:22.360771 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:22.360744    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b3371ba-4d68-47a2-b45d-ef2470346e48-dshm\") pod \"sub-test-model-73435a-kserve-57db64f59-vrxlt\" (UID: \"1b3371ba-4d68-47a2-b45d-ef2470346e48\") " pod="ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt"
Oct 30 01:11:22.360963 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:22.360946    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b3371ba-4d68-47a2-b45d-ef2470346e48-tls-certs\") pod \"sub-test-model-73435a-kserve-57db64f59-vrxlt\" (UID: \"1b3371ba-4d68-47a2-b45d-ef2470346e48\") " pod="ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt"
Oct 30 01:11:22.381763 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:22.381724    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gmj9\" (UniqueName: \"kubernetes.io/projected/1b3371ba-4d68-47a2-b45d-ef2470346e48-kube-api-access-7gmj9\") pod \"sub-test-model-73435a-kserve-57db64f59-vrxlt\" (UID: \"1b3371ba-4d68-47a2-b45d-ef2470346e48\") " pod="ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt"
Oct 30 01:11:22.522509 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:22.522125    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt"
Oct 30 01:11:22.840320 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:11:22.840292    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b3371ba_4d68_47a2_b45d_ef2470346e48.slice/crio-d7b961290840b97e868e5ad6f91f9cf4b3d59d103fb7312d3d6ae8542ef00b46 WatchSource:0}: Error finding container d7b961290840b97e868e5ad6f91f9cf4b3d59d103fb7312d3d6ae8542ef00b46: Status 404 returned error can't find the container with id d7b961290840b97e868e5ad6f91f9cf4b3d59d103fb7312d3d6ae8542ef00b46
Oct 30 01:11:22.840618 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:22.840592    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt"]
Oct 30 01:11:23.273733 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:23.273684    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt" event={"ID":"1b3371ba-4d68-47a2-b45d-ef2470346e48","Type":"ContainerStarted","Data":"178ead6272f08b75482d71381d3b6d6dfa2811d75d8c530769b6208542f13749"}
Oct 30 01:11:23.274217 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:23.273742    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt" event={"ID":"1b3371ba-4d68-47a2-b45d-ef2470346e48","Type":"ContainerStarted","Data":"d7b961290840b97e868e5ad6f91f9cf4b3d59d103fb7312d3d6ae8542ef00b46"}
Oct 30 01:11:23.316208 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:23.316155    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt" podStartSLOduration=1.31614271 podStartE2EDuration="1.31614271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:23.313310069 +0000 UTC m=+2373.700986459" watchObservedRunningTime="2026-10-30 01:11:23.31614271 +0000 UTC m=+2373.703819098"
Oct 30 01:11:23.522836 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:23.522772    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt"
Oct 30 01:11:24.495387 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:24.495357    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8822-4bab5b"
Oct 30 01:11:24.592967 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:24.592885    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b65e76c5-7825-4900-9e59-fae76d726f1f-kube-api-access-lk9wr\" (UniqueName: \"kubernetes.io/projected/b65e76c5-7825-4900-9e59-fae76d726f1f-kube-api-access-lk9wr\") pod \"b65e76c5-7825-4900-9e59-fae76d726f1f\" (UID: \"b65e76c5-7825-4900-9e59-fae76d726f1f\") "
Oct 30 01:11:24.595210 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:24.595178    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b65e76c5-7825-4900-9e59-fae76d726f1f-kube-api-access-lk9wr" pod "b65e76c5-7825-4900-9e59-fae76d726f1f" (UID: "b65e76c5-7825-4900-9e59-fae76d726f1f"). InnerVolumeSpecName "kube-api-access-lk9wr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:11:24.694425 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:24.694396    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk9wr\" (UniqueName: \"kubernetes.io/projected/b65e76c5-7825-4900-9e59-fae76d726f1f-kube-api-access-lk9wr\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:11:25.292360 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:25.292323    4393 generic.go:356] "Generic (PLEG): container finished" podID="b65e76c5-7825-4900-9e59-fae76d726f1f" containerID="d33ce9587964489f2cb846f0d59a860d479e911fa32d6568e17c558e9a3c1d59" exitCode=137
Oct 30 01:11:25.292533 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:25.292388    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8822-4bab5b"
Oct 30 01:11:25.292533 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:25.292424    4393 scope.go:122] "RemoveContainer" containerID="d33ce9587964489f2cb846f0d59a860d479e911fa32d6568e17c558e9a3c1d59"
Oct 30 01:11:25.309213 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:25.309192    4393 scope.go:122] "RemoveContainer" containerID="d33ce9587964489f2cb846f0d59a860d479e911fa32d6568e17c558e9a3c1d59"
Oct 30 01:11:25.309587 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:11:25.309566    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d33ce9587964489f2cb846f0d59a860d479e911fa32d6568e17c558e9a3c1d59\": container with ID starting with d33ce9587964489f2cb846f0d59a860d479e911fa32d6568e17c558e9a3c1d59 not found: ID does not exist" containerID="d33ce9587964489f2cb846f0d59a860d479e911fa32d6568e17c558e9a3c1d59"
Oct 30 01:11:25.309657 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:25.309596    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d33ce9587964489f2cb846f0d59a860d479e911fa32d6568e17c558e9a3c1d59"} err="failed to get container status \"d33ce9587964489f2cb846f0d59a860d479e911fa32d6568e17c558e9a3c1d59\": rpc error: code = NotFound desc = could not find container \"d33ce9587964489f2cb846f0d59a860d479e911fa32d6568e17c558e9a3c1d59\": container with ID starting with d33ce9587964489f2cb846f0d59a860d479e911fa32d6568e17c558e9a3c1d59 not found: ID does not exist"
Oct 30 01:11:25.317788 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:11:25.317748    4393 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8822-4bab5b\" is forbidden: User \"system:node:ip-10-0-135-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-56.ec2.internal' and this object" podUID="b65e76c5-7825-4900-9e59-fae76d726f1f" pod="opendatahub/test-curl-8822-4bab5b"
Oct 30 01:11:25.727185 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:25.727152    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b65e76c5-7825-4900-9e59-fae76d726f1f" path="/var/lib/kubelet/pods/b65e76c5-7825-4900-9e59-fae76d726f1f/volumes"
Oct 30 01:11:26.512130 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:11:26.512096    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:11:28.229905 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:28.229877    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-68746cfdb7-hsxp2"
Oct 30 01:11:32.494965 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:32.494886    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567"]
Oct 30 01:11:32.501114 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:32.501093    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567"
Oct 30 01:11:32.529006 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:32.525293    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt"
Oct 30 01:11:32.529006 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:32.526300    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/473e63de-a201-41a6-b927-eee7643de4e1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567\" (UID: \"473e63de-a201-41a6-b927-eee7643de4e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567"
Oct 30 01:11:32.529006 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:32.526345    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/473e63de-a201-41a6-b927-eee7643de4e1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567\" (UID: \"473e63de-a201-41a6-b927-eee7643de4e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567"
Oct 30 01:11:32.529006 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:32.526388    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6b67\" (UniqueName: \"kubernetes.io/projected/473e63de-a201-41a6-b927-eee7643de4e1-kube-api-access-f6b67\") pod \"maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567\" (UID: \"473e63de-a201-41a6-b927-eee7643de4e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567"
Oct 30 01:11:32.529006 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:32.526411    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/473e63de-a201-41a6-b927-eee7643de4e1-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567\" (UID: \"473e63de-a201-41a6-b927-eee7643de4e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567"
Oct 30 01:11:32.529006 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:32.526643    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567"]
Oct 30 01:11:32.529006 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:32.528076    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-a3619c-serving-cert\""
Oct 30 01:11:32.546467 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:32.546440    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt"
Oct 30 01:11:32.628182 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:32.628145    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/473e63de-a201-41a6-b927-eee7643de4e1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567\" (UID: \"473e63de-a201-41a6-b927-eee7643de4e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567"
Oct 30 01:11:32.628387 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:32.628188    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/473e63de-a201-41a6-b927-eee7643de4e1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567\" (UID: \"473e63de-a201-41a6-b927-eee7643de4e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567"
Oct 30 01:11:32.628387 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:32.628206    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6b67\" (UniqueName: \"kubernetes.io/projected/473e63de-a201-41a6-b927-eee7643de4e1-kube-api-access-f6b67\") pod \"maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567\" (UID: \"473e63de-a201-41a6-b927-eee7643de4e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567"
Oct 30 01:11:32.628387 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:32.628329    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/473e63de-a201-41a6-b927-eee7643de4e1-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567\" (UID: \"473e63de-a201-41a6-b927-eee7643de4e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567"
Oct 30 01:11:32.628903 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:32.628874    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/473e63de-a201-41a6-b927-eee7643de4e1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567\" (UID: \"473e63de-a201-41a6-b927-eee7643de4e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567"
Oct 30 01:11:32.630871 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:32.630851    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/473e63de-a201-41a6-b927-eee7643de4e1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567\" (UID: \"473e63de-a201-41a6-b927-eee7643de4e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567"
Oct 30 01:11:32.631159 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:32.631137    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/473e63de-a201-41a6-b927-eee7643de4e1-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567\" (UID: \"473e63de-a201-41a6-b927-eee7643de4e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567"
Oct 30 01:11:32.651414 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:32.651381    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6b67\" (UniqueName: \"kubernetes.io/projected/473e63de-a201-41a6-b927-eee7643de4e1-kube-api-access-f6b67\") pod \"maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567\" (UID: \"473e63de-a201-41a6-b927-eee7643de4e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567"
Oct 30 01:11:32.817177 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:32.817082    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567"
Oct 30 01:11:33.130149 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:11:33.130113    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod473e63de_a201_41a6_b927_eee7643de4e1.slice/crio-ef86fe8c28126d5ee7e3744a27f4058af2971ab0da7c0de46be925bc3cab0891 WatchSource:0}: Error finding container ef86fe8c28126d5ee7e3744a27f4058af2971ab0da7c0de46be925bc3cab0891: Status 404 returned error can't find the container with id ef86fe8c28126d5ee7e3744a27f4058af2971ab0da7c0de46be925bc3cab0891
Oct 30 01:11:33.130718 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:33.130691    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567"]
Oct 30 01:11:33.369214 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:33.369174    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567" event={"ID":"473e63de-a201-41a6-b927-eee7643de4e1","Type":"ContainerStarted","Data":"ef86fe8c28126d5ee7e3744a27f4058af2971ab0da7c0de46be925bc3cab0891"}
Oct 30 01:11:33.382263 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:33.382188    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt"
Oct 30 01:11:34.390641 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:34.390595    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567" event={"ID":"473e63de-a201-41a6-b927-eee7643de4e1","Type":"ContainerStarted","Data":"4471345e9a58379ec1e35682083ebff8af939809aae778665a5b7ff812fa8ce1"}
Oct 30 01:11:34.438840 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:34.438741    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567" podStartSLOduration=2.081479667 podStartE2EDuration="2.438720657s" totalImagesPullingTime="357.24099ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:34.431015996 +0000 UTC m=+2384.818692385" watchObservedRunningTime="2026-10-30 01:11:34.438720657 +0000 UTC m=+2384.826397047"
Oct 30 01:11:35.398590 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:35.398562    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567"
Oct 30 01:11:36.043841 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:36.043808    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"]
Oct 30 01:11:36.049450 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:36.049429    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"
Oct 30 01:11:36.063693 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:36.063669    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-beeba4-openshift-default-dockercfg-5vtrz\""
Oct 30 01:11:36.088337 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:36.088302    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"]
Oct 30 01:11:36.221898 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:36.221865    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a26137-dc63-40e1-89cc-2a0c70d6f52e-istiod-ca-cert\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8\" (UID: \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"
Oct 30 01:11:36.222110 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:36.221926    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29cm6\" (UniqueName: \"kubernetes.io/projected/40a26137-dc63-40e1-89cc-2a0c70d6f52e-kube-api-access-29cm6\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8\" (UID: \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"
Oct 30 01:11:36.222110 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:36.222000    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a26137-dc63-40e1-89cc-2a0c70d6f52e-workload-socket\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8\" (UID: \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"
Oct 30 01:11:36.222110 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:36.222035    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a26137-dc63-40e1-89cc-2a0c70d6f52e-workload-certs\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8\" (UID: \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"
Oct 30 01:11:36.222110 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:36.222106    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a26137-dc63-40e1-89cc-2a0c70d6f52e-istio-podinfo\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8\" (UID: \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"
Oct 30 01:11:36.222342 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:36.222138    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40a26137-dc63-40e1-89cc-2a0c70d6f52e-istio-token\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8\" (UID: \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"
Oct 30 01:11:36.222342 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:36.222216    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a26137-dc63-40e1-89cc-2a0c70d6f52e-istio-envoy\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8\" (UID: \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"
Oct 30 01:11:36.222342 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:36.222235    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a26137-dc63-40e1-89cc-2a0c70d6f52e-credential-socket\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8\" (UID: \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"
Oct 30 01:11:36.222342 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:36.222320    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a26137-dc63-40e1-89cc-2a0c70d6f52e-istio-data\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8\" (UID: \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"
Oct 30 01:11:36.323256 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:36.323163    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a26137-dc63-40e1-89cc-2a0c70d6f52e-istiod-ca-cert\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8\" (UID: \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"
Oct 30 01:11:36.323256 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:36.323227    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29cm6\" (UniqueName: \"kubernetes.io/projected/40a26137-dc63-40e1-89cc-2a0c70d6f52e-kube-api-access-29cm6\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8\" (UID: \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"
Oct 30 01:11:36.323256 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:36.323248    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a26137-dc63-40e1-89cc-2a0c70d6f52e-workload-socket\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8\" (UID: \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"
Oct 30 01:11:36.323545 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:36.323358    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a26137-dc63-40e1-89cc-2a0c70d6f52e-workload-certs\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8\" (UID: \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"
Oct 30 01:11:36.323545 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:36.323451    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a26137-dc63-40e1-89cc-2a0c70d6f52e-istio-podinfo\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8\" (UID: \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"
Oct 30 01:11:36.323545 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:36.323479    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40a26137-dc63-40e1-89cc-2a0c70d6f52e-istio-token\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8\" (UID: \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"
Oct 30 01:11:36.323718 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:36.323575    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a26137-dc63-40e1-89cc-2a0c70d6f52e-istio-envoy\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8\" (UID: \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"
Oct 30 01:11:36.323718 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:36.323606    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a26137-dc63-40e1-89cc-2a0c70d6f52e-credential-socket\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8\" (UID: \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"
Oct 30 01:11:36.323946 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:36.323622    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a26137-dc63-40e1-89cc-2a0c70d6f52e-workload-socket\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8\" (UID: \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"
Oct 30 01:11:36.324035 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:36.323925    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a26137-dc63-40e1-89cc-2a0c70d6f52e-istio-data\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8\" (UID: \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"
Oct 30 01:11:36.324035 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:36.323964    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a26137-dc63-40e1-89cc-2a0c70d6f52e-workload-certs\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8\" (UID: \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"
Oct 30 01:11:36.324035 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:36.323992    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a26137-dc63-40e1-89cc-2a0c70d6f52e-credential-socket\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8\" (UID: \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"
Oct 30 01:11:36.324167 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:36.324064    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a26137-dc63-40e1-89cc-2a0c70d6f52e-istiod-ca-cert\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8\" (UID: \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"
Oct 30 01:11:36.324167 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:36.324147    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a26137-dc63-40e1-89cc-2a0c70d6f52e-istio-data\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8\" (UID: \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"
Oct 30 01:11:36.325967 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:36.325936    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a26137-dc63-40e1-89cc-2a0c70d6f52e-istio-envoy\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8\" (UID: \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"
Oct 30 01:11:36.326312 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:36.326293    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a26137-dc63-40e1-89cc-2a0c70d6f52e-istio-podinfo\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8\" (UID: \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"
Oct 30 01:11:36.345656 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:36.345633    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40a26137-dc63-40e1-89cc-2a0c70d6f52e-istio-token\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8\" (UID: \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"
Oct 30 01:11:36.346028 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:36.346006    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29cm6\" (UniqueName: \"kubernetes.io/projected/40a26137-dc63-40e1-89cc-2a0c70d6f52e-kube-api-access-29cm6\") pod \"e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8\" (UID: \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\") " pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"
Oct 30 01:11:36.366517 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:36.366483    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"
Oct 30 01:11:36.697006 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:11:36.696972    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40a26137_dc63_40e1_89cc_2a0c70d6f52e.slice/crio-016f617a4502c4dfb77692a0d4ffd5192a23a498c25491be1be6a178387e2f81 WatchSource:0}: Error finding container 016f617a4502c4dfb77692a0d4ffd5192a23a498c25491be1be6a178387e2f81: Status 404 returned error can't find the container with id 016f617a4502c4dfb77692a0d4ffd5192a23a498c25491be1be6a178387e2f81
Oct 30 01:11:36.697841 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:36.697810    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"]
Oct 30 01:11:36.700248 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:36.700212    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:11:36.700358 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:36.700278    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:11:36.700358 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:36.700308    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:11:37.426806 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:37.426250    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8" event={"ID":"40a26137-dc63-40e1-89cc-2a0c70d6f52e","Type":"ContainerStarted","Data":"ef2bacbded603aa589533f089fe22a99843fd42bda70befcccd7934d258d89cd"}
Oct 30 01:11:37.426806 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:37.426357    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8" event={"ID":"40a26137-dc63-40e1-89cc-2a0c70d6f52e","Type":"ContainerStarted","Data":"016f617a4502c4dfb77692a0d4ffd5192a23a498c25491be1be6a178387e2f81"}
Oct 30 01:11:37.493103 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:37.493052    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8" podStartSLOduration=1.493039225 podStartE2EDuration="1.493039225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:37.480932516 +0000 UTC m=+2387.868608905" watchObservedRunningTime="2026-10-30 01:11:37.493039225 +0000 UTC m=+2387.880715613"
Oct 30 01:11:38.331061 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:38.331021    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq"]
Oct 30 01:11:38.336094 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:38.336073    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq"
Oct 30 01:11:38.350535 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:38.350510    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq"]
Oct 30 01:11:38.372188 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:38.372165    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-a7ca1f-serving-cert\""
Oct 30 01:11:38.373266 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:38.373250    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"
Oct 30 01:11:38.373470 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:38.373438    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f10a91a2-6575-46ff-a98d-73317cf11bfc-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq\" (UID: \"f10a91a2-6575-46ff-a98d-73317cf11bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq"
Oct 30 01:11:38.373529 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:38.373508    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97tjm\" (UniqueName: \"kubernetes.io/projected/f10a91a2-6575-46ff-a98d-73317cf11bfc-kube-api-access-97tjm\") pod \"maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq\" (UID: \"f10a91a2-6575-46ff-a98d-73317cf11bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq"
Oct 30 01:11:38.373569 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:38.373550    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f10a91a2-6575-46ff-a98d-73317cf11bfc-metrics-certs\") pod \"maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq\" (UID: \"f10a91a2-6575-46ff-a98d-73317cf11bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq"
Oct 30 01:11:38.373633 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:38.373615    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f10a91a2-6575-46ff-a98d-73317cf11bfc-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq\" (UID: \"f10a91a2-6575-46ff-a98d-73317cf11bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq"
Oct 30 01:11:38.374585 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:38.374560    4393 patch_prober.go:29] interesting pod/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 01:11:38.374680 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:38.374612    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8" podUID="40a26137-dc63-40e1-89cc-2a0c70d6f52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 01:11:38.475388 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:38.474971    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f10a91a2-6575-46ff-a98d-73317cf11bfc-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq\" (UID: \"f10a91a2-6575-46ff-a98d-73317cf11bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq"
Oct 30 01:11:38.475388 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:38.475021    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97tjm\" (UniqueName: \"kubernetes.io/projected/f10a91a2-6575-46ff-a98d-73317cf11bfc-kube-api-access-97tjm\") pod \"maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq\" (UID: \"f10a91a2-6575-46ff-a98d-73317cf11bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq"
Oct 30 01:11:38.475388 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:38.475047    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f10a91a2-6575-46ff-a98d-73317cf11bfc-metrics-certs\") pod \"maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq\" (UID: \"f10a91a2-6575-46ff-a98d-73317cf11bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq"
Oct 30 01:11:38.475388 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:38.475095    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f10a91a2-6575-46ff-a98d-73317cf11bfc-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq\" (UID: \"f10a91a2-6575-46ff-a98d-73317cf11bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq"
Oct 30 01:11:38.475388 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:11:38.475274    4393 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7ca1f-serving-cert: secret "maas-api-e2e-ipp-a-a7ca1f-serving-cert" not found
Oct 30 01:11:38.475388 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:11:38.475318    4393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f10a91a2-6575-46ff-a98d-73317cf11bfc-maas-api-tls podName:f10a91a2-6575-46ff-a98d-73317cf11bfc nodeName:}" failed. No retries permitted until 2026-10-30 01:11:38.97530502 +0000 UTC m=+2389.362981388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f10a91a2-6575-46ff-a98d-73317cf11bfc-maas-api-tls") pod "maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq" (UID: "f10a91a2-6575-46ff-a98d-73317cf11bfc") : secret "maas-api-e2e-ipp-a-a7ca1f-serving-cert" not found
Oct 30 01:11:38.476079 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:38.476056    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f10a91a2-6575-46ff-a98d-73317cf11bfc-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq\" (UID: \"f10a91a2-6575-46ff-a98d-73317cf11bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq"
Oct 30 01:11:38.478314 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:38.478284    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f10a91a2-6575-46ff-a98d-73317cf11bfc-metrics-certs\") pod \"maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq\" (UID: \"f10a91a2-6575-46ff-a98d-73317cf11bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq"
Oct 30 01:11:38.500350 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:38.500321    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97tjm\" (UniqueName: \"kubernetes.io/projected/f10a91a2-6575-46ff-a98d-73317cf11bfc-kube-api-access-97tjm\") pod \"maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq\" (UID: \"f10a91a2-6575-46ff-a98d-73317cf11bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq"
Oct 30 01:11:38.985231 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:38.985189    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f10a91a2-6575-46ff-a98d-73317cf11bfc-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq\" (UID: \"f10a91a2-6575-46ff-a98d-73317cf11bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq"
Oct 30 01:11:38.985452 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:11:38.985360    4393 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7ca1f-serving-cert: secret "maas-api-e2e-ipp-a-a7ca1f-serving-cert" not found
Oct 30 01:11:38.985452 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:11:38.985442    4393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f10a91a2-6575-46ff-a98d-73317cf11bfc-maas-api-tls podName:f10a91a2-6575-46ff-a98d-73317cf11bfc nodeName:}" failed. No retries permitted until 2026-10-30 01:11:39.98542261 +0000 UTC m=+2390.373098977 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f10a91a2-6575-46ff-a98d-73317cf11bfc-maas-api-tls") pod "maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq" (UID: "f10a91a2-6575-46ff-a98d-73317cf11bfc") : secret "maas-api-e2e-ipp-a-a7ca1f-serving-cert" not found
Oct 30 01:11:39.367234 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:39.367137    4393 patch_prober.go:29] interesting pod/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 01:11:39.367234 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:39.367213    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8" podUID="40a26137-dc63-40e1-89cc-2a0c70d6f52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 01:11:40.001500 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:40.001449    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f10a91a2-6575-46ff-a98d-73317cf11bfc-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq\" (UID: \"f10a91a2-6575-46ff-a98d-73317cf11bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq"
Oct 30 01:11:40.030424 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:40.030388    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f10a91a2-6575-46ff-a98d-73317cf11bfc-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq\" (UID: \"f10a91a2-6575-46ff-a98d-73317cf11bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq"
Oct 30 01:11:40.154002 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:40.153972    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq"
Oct 30 01:11:40.367166 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:40.367064    4393 patch_prober.go:29] interesting pod/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 01:11:40.367166 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:40.367133    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8" podUID="40a26137-dc63-40e1-89cc-2a0c70d6f52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 01:11:40.486861 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:40.486833    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq"]
Oct 30 01:11:40.488602 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:11:40.488569    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf10a91a2_6575_46ff_a98d_73317cf11bfc.slice/crio-f7046009f8efd6c3bdb69ea0fe68a522617a6b6edbc627a21c524d223012a17c WatchSource:0}: Error finding container f7046009f8efd6c3bdb69ea0fe68a522617a6b6edbc627a21c524d223012a17c: Status 404 returned error can't find the container with id f7046009f8efd6c3bdb69ea0fe68a522617a6b6edbc627a21c524d223012a17c
Oct 30 01:11:41.367144 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:41.367109    4393 patch_prober.go:29] interesting pod/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 01:11:41.367318 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:41.367157    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8" podUID="40a26137-dc63-40e1-89cc-2a0c70d6f52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 01:11:41.431072 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:41.431018    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567"
Oct 30 01:11:41.468279 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:41.468244    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq" event={"ID":"f10a91a2-6575-46ff-a98d-73317cf11bfc","Type":"ContainerStarted","Data":"a45d15ab337fe16c1a7cf8bf6948ff118f5e06e3b0b71bdfe25358e8fcdaaafb"}
Oct 30 01:11:41.468501 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:41.468295    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq" event={"ID":"f10a91a2-6575-46ff-a98d-73317cf11bfc","Type":"ContainerStarted","Data":"f7046009f8efd6c3bdb69ea0fe68a522617a6b6edbc627a21c524d223012a17c"}
Oct 30 01:11:41.468501 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:41.468322    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq"
Oct 30 01:11:41.536307 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:41.536238    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq" podStartSLOduration=3.180060408 podStartE2EDuration="3.536217471s" totalImagesPullingTime="356.157063ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:41.530246249 +0000 UTC m=+2391.917922661" watchObservedRunningTime="2026-10-30 01:11:41.536217471 +0000 UTC m=+2391.923893861"
Oct 30 01:11:42.366903 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:42.366868    4393 patch_prober.go:29] interesting pod/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 01:11:42.367081 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:42.366923    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8" podUID="40a26137-dc63-40e1-89cc-2a0c70d6f52e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 01:11:43.371707 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:43.371676    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"
Oct 30 01:11:43.372234 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:43.372212    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"
Oct 30 01:11:43.373136 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:43.373113    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"
Oct 30 01:11:44.113449 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:44.113414    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt"]
Oct 30 01:11:44.118582 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:44.118561    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt"
Oct 30 01:11:44.138369 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:44.138341    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt"]
Oct 30 01:11:44.146709 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:44.146682    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-d8b2c6-serving-cert\""
Oct 30 01:11:44.203272 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:44.203241    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecb5a5fb-3709-40c0-9ae4-b699d51ec79d-metrics-certs\") pod \"maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt\" (UID: \"ecb5a5fb-3709-40c0-9ae4-b699d51ec79d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt"
Oct 30 01:11:44.203441 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:44.203293    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecb5a5fb-3709-40c0-9ae4-b699d51ec79d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt\" (UID: \"ecb5a5fb-3709-40c0-9ae4-b699d51ec79d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt"
Oct 30 01:11:44.203441 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:44.203421    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpkrb\" (UniqueName: \"kubernetes.io/projected/ecb5a5fb-3709-40c0-9ae4-b699d51ec79d-kube-api-access-jpkrb\") pod \"maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt\" (UID: \"ecb5a5fb-3709-40c0-9ae4-b699d51ec79d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt"
Oct 30 01:11:44.203517 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:44.203458    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecb5a5fb-3709-40c0-9ae4-b699d51ec79d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt\" (UID: \"ecb5a5fb-3709-40c0-9ae4-b699d51ec79d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt"
Oct 30 01:11:44.317814 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:44.309459    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecb5a5fb-3709-40c0-9ae4-b699d51ec79d-metrics-certs\") pod \"maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt\" (UID: \"ecb5a5fb-3709-40c0-9ae4-b699d51ec79d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt"
Oct 30 01:11:44.334158 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:44.334120    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecb5a5fb-3709-40c0-9ae4-b699d51ec79d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt\" (UID: \"ecb5a5fb-3709-40c0-9ae4-b699d51ec79d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt"
Oct 30 01:11:44.334372 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:44.334308    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpkrb\" (UniqueName: \"kubernetes.io/projected/ecb5a5fb-3709-40c0-9ae4-b699d51ec79d-kube-api-access-jpkrb\") pod \"maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt\" (UID: \"ecb5a5fb-3709-40c0-9ae4-b699d51ec79d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt"
Oct 30 01:11:44.334372 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:44.334364    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecb5a5fb-3709-40c0-9ae4-b699d51ec79d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt\" (UID: \"ecb5a5fb-3709-40c0-9ae4-b699d51ec79d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt"
Oct 30 01:11:44.336255 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:44.336227    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecb5a5fb-3709-40c0-9ae4-b699d51ec79d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt\" (UID: \"ecb5a5fb-3709-40c0-9ae4-b699d51ec79d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt"
Oct 30 01:11:44.366695 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:44.362683    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecb5a5fb-3709-40c0-9ae4-b699d51ec79d-metrics-certs\") pod \"maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt\" (UID: \"ecb5a5fb-3709-40c0-9ae4-b699d51ec79d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt"
Oct 30 01:11:44.366695 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:44.364440    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpkrb\" (UniqueName: \"kubernetes.io/projected/ecb5a5fb-3709-40c0-9ae4-b699d51ec79d-kube-api-access-jpkrb\") pod \"maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt\" (UID: \"ecb5a5fb-3709-40c0-9ae4-b699d51ec79d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt"
Oct 30 01:11:44.366695 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:44.365632    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecb5a5fb-3709-40c0-9ae4-b699d51ec79d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt\" (UID: \"ecb5a5fb-3709-40c0-9ae4-b699d51ec79d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt"
Oct 30 01:11:44.434726 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:44.434687    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt"
Oct 30 01:11:44.762217 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:11:44.762183    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecb5a5fb_3709_40c0_9ae4_b699d51ec79d.slice/crio-df21017f86179e2bed2a35a568d03e86a57bbd9fe65cbd36ec19cdf3e3f3b18f WatchSource:0}: Error finding container df21017f86179e2bed2a35a568d03e86a57bbd9fe65cbd36ec19cdf3e3f3b18f: Status 404 returned error can't find the container with id df21017f86179e2bed2a35a568d03e86a57bbd9fe65cbd36ec19cdf3e3f3b18f
Oct 30 01:11:44.762443 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:44.762349    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt"]
Oct 30 01:11:45.528306 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:45.528214    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt" event={"ID":"ecb5a5fb-3709-40c0-9ae4-b699d51ec79d","Type":"ContainerStarted","Data":"0ca5792c2281a7c4824c839d7d5b349495965e2896c8400cdfd1cac5ca1374ec"}
Oct 30 01:11:45.528306 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:45.528285    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt" event={"ID":"ecb5a5fb-3709-40c0-9ae4-b699d51ec79d","Type":"ContainerStarted","Data":"df21017f86179e2bed2a35a568d03e86a57bbd9fe65cbd36ec19cdf3e3f3b18f"}
Oct 30 01:11:45.528924 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:45.528328    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt"
Oct 30 01:11:45.571033 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:45.570981    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt" podStartSLOduration=1.192081451 podStartE2EDuration="1.57096605s" totalImagesPullingTime="378.884599ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:11:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:11:45.568873444 +0000 UTC m=+2395.956549830" watchObservedRunningTime="2026-10-30 01:11:45.57096605 +0000 UTC m=+2395.958642439"
Oct 30 01:11:47.482283 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:47.482253    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq"
Oct 30 01:11:51.543150 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:11:51.543120    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt"
Oct 30 01:11:56.552920 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:11:56.545495    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:12:13.523559 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:13.523526    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7"]
Oct 30 01:12:20.215421 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:20.215386    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj"]
Oct 30 01:12:20.220540 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:20.220521    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj"
Oct 30 01:12:20.238875 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:20.238846    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj"]
Oct 30 01:12:20.258139 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:20.258100    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-6d44c2-serving-cert\""
Oct 30 01:12:20.301542 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:20.301508    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0dd37a5-2f69-4f95-9e3e-958735302616-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj\" (UID: \"a0dd37a5-2f69-4f95-9e3e-958735302616\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj"
Oct 30 01:12:20.301863 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:20.301838    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0dd37a5-2f69-4f95-9e3e-958735302616-metrics-certs\") pod \"maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj\" (UID: \"a0dd37a5-2f69-4f95-9e3e-958735302616\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj"
Oct 30 01:12:20.301938 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:20.301899    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0dd37a5-2f69-4f95-9e3e-958735302616-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj\" (UID: \"a0dd37a5-2f69-4f95-9e3e-958735302616\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj"
Oct 30 01:12:20.301987 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:20.301973    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v8jg\" (UniqueName: \"kubernetes.io/projected/a0dd37a5-2f69-4f95-9e3e-958735302616-kube-api-access-5v8jg\") pod \"maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj\" (UID: \"a0dd37a5-2f69-4f95-9e3e-958735302616\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj"
Oct 30 01:12:20.403629 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:20.403588    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0dd37a5-2f69-4f95-9e3e-958735302616-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj\" (UID: \"a0dd37a5-2f69-4f95-9e3e-958735302616\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj"
Oct 30 01:12:20.403903 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:20.403884    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0dd37a5-2f69-4f95-9e3e-958735302616-metrics-certs\") pod \"maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj\" (UID: \"a0dd37a5-2f69-4f95-9e3e-958735302616\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj"
Oct 30 01:12:20.404011 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:20.403934    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0dd37a5-2f69-4f95-9e3e-958735302616-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj\" (UID: \"a0dd37a5-2f69-4f95-9e3e-958735302616\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj"
Oct 30 01:12:20.404073 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:20.404015    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v8jg\" (UniqueName: \"kubernetes.io/projected/a0dd37a5-2f69-4f95-9e3e-958735302616-kube-api-access-5v8jg\") pod \"maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj\" (UID: \"a0dd37a5-2f69-4f95-9e3e-958735302616\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj"
Oct 30 01:12:20.404652 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:20.404631    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0dd37a5-2f69-4f95-9e3e-958735302616-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj\" (UID: \"a0dd37a5-2f69-4f95-9e3e-958735302616\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj"
Oct 30 01:12:20.406430 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:20.406409    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0dd37a5-2f69-4f95-9e3e-958735302616-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj\" (UID: \"a0dd37a5-2f69-4f95-9e3e-958735302616\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj"
Oct 30 01:12:20.406523 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:20.406448    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0dd37a5-2f69-4f95-9e3e-958735302616-metrics-certs\") pod \"maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj\" (UID: \"a0dd37a5-2f69-4f95-9e3e-958735302616\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj"
Oct 30 01:12:20.427567 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:20.427537    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v8jg\" (UniqueName: \"kubernetes.io/projected/a0dd37a5-2f69-4f95-9e3e-958735302616-kube-api-access-5v8jg\") pod \"maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj\" (UID: \"a0dd37a5-2f69-4f95-9e3e-958735302616\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj"
Oct 30 01:12:20.539329 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:20.539237    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj"
Oct 30 01:12:20.866009 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:12:20.865976    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0dd37a5_2f69_4f95_9e3e_958735302616.slice/crio-1933f14665e937fb37d391d1219a2aa18907f462e0008696ae32a39f99bf6d7e WatchSource:0}: Error finding container 1933f14665e937fb37d391d1219a2aa18907f462e0008696ae32a39f99bf6d7e: Status 404 returned error can't find the container with id 1933f14665e937fb37d391d1219a2aa18907f462e0008696ae32a39f99bf6d7e
Oct 30 01:12:20.866210 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:20.866187    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj"]
Oct 30 01:12:20.884691 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:20.884661    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj" event={"ID":"a0dd37a5-2f69-4f95-9e3e-958735302616","Type":"ContainerStarted","Data":"1933f14665e937fb37d391d1219a2aa18907f462e0008696ae32a39f99bf6d7e"}
Oct 30 01:12:21.895370 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:21.895335    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj" event={"ID":"a0dd37a5-2f69-4f95-9e3e-958735302616","Type":"ContainerStarted","Data":"710c7da91e5cedc6165605df2b48fa56bd928b0b2a2bf4a07e7d7ee19b6b9e14"}
Oct 30 01:12:21.895797 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:21.895409    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj"
Oct 30 01:12:21.940886 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:21.940829    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj" podStartSLOduration=1.60835248 podStartE2EDuration="1.940816228s" totalImagesPullingTime="332.463748ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:21.935665978 +0000 UTC m=+2432.323342367" watchObservedRunningTime="2026-10-30 01:12:21.940816228 +0000 UTC m=+2432.328492616"
Oct 30 01:12:26.512536 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:12:26.512496    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:12:27.908340 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:27.908307    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj"
Oct 30 01:12:28.254314 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.254226    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"]
Oct 30 01:12:28.262550 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.262522    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"
Oct 30 01:12:28.276218 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.276189    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"]
Oct 30 01:12:28.281266 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.281238    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-73435a-openshift-default-dockercfg-whl5c\""
Oct 30 01:12:28.338562 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.338511    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f083e40-d31b-4313-8459-f5ab9ac9d743-workload-certs\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz\" (UID: \"9f083e40-d31b-4313-8459-f5ab9ac9d743\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"
Oct 30 01:12:28.338562 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.338569    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f083e40-d31b-4313-8459-f5ab9ac9d743-istio-podinfo\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz\" (UID: \"9f083e40-d31b-4313-8459-f5ab9ac9d743\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"
Oct 30 01:12:28.338815 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.338697    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z47q\" (UniqueName: \"kubernetes.io/projected/9f083e40-d31b-4313-8459-f5ab9ac9d743-kube-api-access-6z47q\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz\" (UID: \"9f083e40-d31b-4313-8459-f5ab9ac9d743\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"
Oct 30 01:12:28.338815 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.338736    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f083e40-d31b-4313-8459-f5ab9ac9d743-workload-socket\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz\" (UID: \"9f083e40-d31b-4313-8459-f5ab9ac9d743\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"
Oct 30 01:12:28.338906 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.338836    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f083e40-d31b-4313-8459-f5ab9ac9d743-istio-token\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz\" (UID: \"9f083e40-d31b-4313-8459-f5ab9ac9d743\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"
Oct 30 01:12:28.338906 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.338891    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f083e40-d31b-4313-8459-f5ab9ac9d743-istiod-ca-cert\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz\" (UID: \"9f083e40-d31b-4313-8459-f5ab9ac9d743\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"
Oct 30 01:12:28.338974 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.338917    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f083e40-d31b-4313-8459-f5ab9ac9d743-credential-socket\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz\" (UID: \"9f083e40-d31b-4313-8459-f5ab9ac9d743\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"
Oct 30 01:12:28.339043 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.339013    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f083e40-d31b-4313-8459-f5ab9ac9d743-istio-data\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz\" (UID: \"9f083e40-d31b-4313-8459-f5ab9ac9d743\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"
Oct 30 01:12:28.339086 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.339074    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f083e40-d31b-4313-8459-f5ab9ac9d743-istio-envoy\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz\" (UID: \"9f083e40-d31b-4313-8459-f5ab9ac9d743\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"
Oct 30 01:12:28.440174 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.440135    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f083e40-d31b-4313-8459-f5ab9ac9d743-istiod-ca-cert\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz\" (UID: \"9f083e40-d31b-4313-8459-f5ab9ac9d743\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"
Oct 30 01:12:28.440174 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.440179    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f083e40-d31b-4313-8459-f5ab9ac9d743-credential-socket\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz\" (UID: \"9f083e40-d31b-4313-8459-f5ab9ac9d743\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"
Oct 30 01:12:28.440438 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.440229    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f083e40-d31b-4313-8459-f5ab9ac9d743-istio-data\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz\" (UID: \"9f083e40-d31b-4313-8459-f5ab9ac9d743\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"
Oct 30 01:12:28.440438 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.440264    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f083e40-d31b-4313-8459-f5ab9ac9d743-istio-envoy\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz\" (UID: \"9f083e40-d31b-4313-8459-f5ab9ac9d743\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"
Oct 30 01:12:28.440438 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.440395    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f083e40-d31b-4313-8459-f5ab9ac9d743-workload-certs\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz\" (UID: \"9f083e40-d31b-4313-8459-f5ab9ac9d743\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"
Oct 30 01:12:28.440438 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.440411    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f083e40-d31b-4313-8459-f5ab9ac9d743-istio-podinfo\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz\" (UID: \"9f083e40-d31b-4313-8459-f5ab9ac9d743\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"
Oct 30 01:12:28.440666 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.440456    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z47q\" (UniqueName: \"kubernetes.io/projected/9f083e40-d31b-4313-8459-f5ab9ac9d743-kube-api-access-6z47q\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz\" (UID: \"9f083e40-d31b-4313-8459-f5ab9ac9d743\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"
Oct 30 01:12:28.440666 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.440489    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f083e40-d31b-4313-8459-f5ab9ac9d743-workload-socket\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz\" (UID: \"9f083e40-d31b-4313-8459-f5ab9ac9d743\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"
Oct 30 01:12:28.440666 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.440548    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f083e40-d31b-4313-8459-f5ab9ac9d743-istio-token\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz\" (UID: \"9f083e40-d31b-4313-8459-f5ab9ac9d743\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"
Oct 30 01:12:28.440857 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.440662    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f083e40-d31b-4313-8459-f5ab9ac9d743-credential-socket\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz\" (UID: \"9f083e40-d31b-4313-8459-f5ab9ac9d743\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"
Oct 30 01:12:28.440857 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.440681    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f083e40-d31b-4313-8459-f5ab9ac9d743-istio-data\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz\" (UID: \"9f083e40-d31b-4313-8459-f5ab9ac9d743\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"
Oct 30 01:12:28.440857 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.440802    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f083e40-d31b-4313-8459-f5ab9ac9d743-workload-certs\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz\" (UID: \"9f083e40-d31b-4313-8459-f5ab9ac9d743\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"
Oct 30 01:12:28.441019 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.440992    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f083e40-d31b-4313-8459-f5ab9ac9d743-istiod-ca-cert\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz\" (UID: \"9f083e40-d31b-4313-8459-f5ab9ac9d743\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"
Oct 30 01:12:28.441077 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.441017    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f083e40-d31b-4313-8459-f5ab9ac9d743-workload-socket\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz\" (UID: \"9f083e40-d31b-4313-8459-f5ab9ac9d743\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"
Oct 30 01:12:28.442975 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.442951    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f083e40-d31b-4313-8459-f5ab9ac9d743-istio-envoy\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz\" (UID: \"9f083e40-d31b-4313-8459-f5ab9ac9d743\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"
Oct 30 01:12:28.443562 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.443537    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f083e40-d31b-4313-8459-f5ab9ac9d743-istio-podinfo\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz\" (UID: \"9f083e40-d31b-4313-8459-f5ab9ac9d743\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"
Oct 30 01:12:28.463558 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.463530    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f083e40-d31b-4313-8459-f5ab9ac9d743-istio-token\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz\" (UID: \"9f083e40-d31b-4313-8459-f5ab9ac9d743\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"
Oct 30 01:12:28.463731 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.463714    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z47q\" (UniqueName: \"kubernetes.io/projected/9f083e40-d31b-4313-8459-f5ab9ac9d743-kube-api-access-6z47q\") pod \"e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz\" (UID: \"9f083e40-d31b-4313-8459-f5ab9ac9d743\") " pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"
Oct 30 01:12:28.546403 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.546320    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7" podUID="ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4" containerName="main" containerID="cri-o://0b798ce7e605dbcbd34c34afc1ee35d32f984e1e864f5166b89165c12bdb3cbe" gracePeriod=45
Oct 30 01:12:28.585105 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.585082    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"
Oct 30 01:12:28.954335 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.947879    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"]
Oct 30 01:12:28.979098 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.975884    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7"
Oct 30 01:12:28.982814 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.979151    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:12:28.982814 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.979988    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:12:28.982814 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:28.980029    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:12:29.015605 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:29.015085    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz" event={"ID":"9f083e40-d31b-4313-8459-f5ab9ac9d743","Type":"ContainerStarted","Data":"1897290d1fc7439721c504fd4595596ef607edb87ea70479941ff059a3a98967"}
Oct 30 01:12:29.018148 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:29.018119    4393 generic.go:356] "Generic (PLEG): container finished" podID="ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4" containerID="0b798ce7e605dbcbd34c34afc1ee35d32f984e1e864f5166b89165c12bdb3cbe" exitCode=0
Oct 30 01:12:29.018398 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:29.018373    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7" event={"ID":"ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4","Type":"ContainerDied","Data":"0b798ce7e605dbcbd34c34afc1ee35d32f984e1e864f5166b89165c12bdb3cbe"}
Oct 30 01:12:29.018548 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:29.018530    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7" event={"ID":"ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4","Type":"ContainerDied","Data":"588dd7c38b70c376a1bbcfb1c135234ebc765e583a10e918f350136d127d8d1c"}
Oct 30 01:12:29.018672 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:29.018658    4393 scope.go:122] "RemoveContainer" containerID="0b798ce7e605dbcbd34c34afc1ee35d32f984e1e864f5166b89165c12bdb3cbe"
Oct 30 01:12:29.018946 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:29.018929    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7"
Oct 30 01:12:29.048722 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:29.046747    4393 scope.go:122] "RemoveContainer" containerID="0b798ce7e605dbcbd34c34afc1ee35d32f984e1e864f5166b89165c12bdb3cbe"
Oct 30 01:12:29.049254 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:12:29.048963    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b798ce7e605dbcbd34c34afc1ee35d32f984e1e864f5166b89165c12bdb3cbe\": container with ID starting with 0b798ce7e605dbcbd34c34afc1ee35d32f984e1e864f5166b89165c12bdb3cbe not found: ID does not exist" containerID="0b798ce7e605dbcbd34c34afc1ee35d32f984e1e864f5166b89165c12bdb3cbe"
Oct 30 01:12:29.049254 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:29.049006    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b798ce7e605dbcbd34c34afc1ee35d32f984e1e864f5166b89165c12bdb3cbe"} err="failed to get container status \"0b798ce7e605dbcbd34c34afc1ee35d32f984e1e864f5166b89165c12bdb3cbe\": rpc error: code = NotFound desc = could not find container \"0b798ce7e605dbcbd34c34afc1ee35d32f984e1e864f5166b89165c12bdb3cbe\": container with ID starting with 0b798ce7e605dbcbd34c34afc1ee35d32f984e1e864f5166b89165c12bdb3cbe not found: ID does not exist"
Oct 30 01:12:29.057663 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:29.057633    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-tmp-dir\") pod \"ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4\" (UID: \"ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4\") "
Oct 30 01:12:29.057823 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:29.057800    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-model-cache\") pod \"ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4\" (UID: \"ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4\") "
Oct 30 01:12:29.057965 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:29.057946    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-home\" (UniqueName: \"kubernetes.io/empty-dir/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-home\") pod \"ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4\" (UID: \"ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4\") "
Oct 30 01:12:29.058040 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:29.058003    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-kube-api-access-v72lr\" (UniqueName: \"kubernetes.io/projected/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-kube-api-access-v72lr\") pod \"ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4\" (UID: \"ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4\") "
Oct 30 01:12:29.058040 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:29.058032    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-tls-certs\") pod \"ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4\" (UID: \"ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4\") "
Oct 30 01:12:29.058191 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:29.058175    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-dshm\") pod \"ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4\" (UID: \"ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4\") "
Oct 30 01:12:29.060147 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:29.060094    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-model-cache" pod "ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4" (UID: "ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:29.062932 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:29.062882    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-tmp-dir" pod "ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4" (UID: "ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:29.065938 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:29.065909    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-dshm" pod "ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4" (UID: "ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:29.066279 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:29.066258    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-tls-certs" pod "ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4" (UID: "ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:12:29.069817 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:29.067178    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-kube-api-access-v72lr" pod "ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4" (UID: "ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4"). InnerVolumeSpecName "kube-api-access-v72lr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:12:29.069899 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:29.069882    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-home" pod "ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4" (UID: "ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:12:29.203117 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:29.203029    4393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-home\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:12:29.203552 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:29.203525    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v72lr\" (UniqueName: \"kubernetes.io/projected/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-kube-api-access-v72lr\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:12:29.203768 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:29.203749    4393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-tls-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:12:29.203918 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:29.203904    4393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-dshm\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:12:29.204017 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:29.204006    4393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-tmp-dir\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:12:29.204102 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:29.204092    4393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4-model-cache\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:12:29.367251 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:29.367214    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7"]
Oct 30 01:12:29.378757 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:29.378722    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-7f92f6/auth-test-model-7f92f6-kserve-5d69666869-t2hc7"]
Oct 30 01:12:29.743562 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:29.743463    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4" path="/var/lib/kubelet/pods/ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4/volumes"
Oct 30 01:12:30.038165 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:30.038071    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz" event={"ID":"9f083e40-d31b-4313-8459-f5ab9ac9d743","Type":"ContainerStarted","Data":"dbdef56b4ff8c0a3478dbb8dc380ffd7b4948773d125c3b9135109d3ff5bb4be"}
Oct 30 01:12:30.094691 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:30.094641    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz" podStartSLOduration=2.094610498 podStartE2EDuration="2.094610498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:12:30.092614547 +0000 UTC m=+2440.480290937" watchObservedRunningTime="2026-10-30 01:12:30.094610498 +0000 UTC m=+2440.482286886"
Oct 30 01:12:30.585559 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:30.585523    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"
Oct 30 01:12:30.587581 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:30.587556    4393 patch_prober.go:29] interesting pod/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 01:12:30.587711 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:30.587605    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz" podUID="9f083e40-d31b-4313-8459-f5ab9ac9d743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 01:12:31.586264 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:31.586235    4393 patch_prober.go:29] interesting pod/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 01:12:31.586733 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:31.586283    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz" podUID="9f083e40-d31b-4313-8459-f5ab9ac9d743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 01:12:32.586462 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:32.586418    4393 patch_prober.go:29] interesting pod/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 01:12:32.586863 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:32.586489    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz" podUID="9f083e40-d31b-4313-8459-f5ab9ac9d743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 01:12:33.586108 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:33.586071    4393 patch_prober.go:29] interesting pod/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 01:12:33.586292 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:33.586129    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz" podUID="9f083e40-d31b-4313-8459-f5ab9ac9d743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 01:12:34.587803 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:34.586183    4393 patch_prober.go:29] interesting pod/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 30 01:12:34.587803 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:34.586248    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz" podUID="9f083e40-d31b-4313-8459-f5ab9ac9d743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 30 01:12:35.791308 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:35.791274    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"
Oct 30 01:12:35.791850 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:35.791818    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"
Oct 30 01:12:35.792730 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:12:35.792707    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"
Oct 30 01:12:56.513409 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:12:56.513366    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:13:24.661855 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:24.661816    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf"]
Oct 30 01:13:26.512008 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:13:26.511974    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:13:39.733179 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:39.733142    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf" podUID="d7a01fa7-260e-466c-b416-aad24c892577" containerName="main" containerID="cri-o://45fd3fd7fd48861b334246daf47e6084e429dd675eb5c3f8ecc5466a4e3259e8" gracePeriod=45
Oct 30 01:13:40.025453 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:40.025416    4393 generic.go:356] "Generic (PLEG): container finished" podID="d7a01fa7-260e-466c-b416-aad24c892577" containerID="45fd3fd7fd48861b334246daf47e6084e429dd675eb5c3f8ecc5466a4e3259e8" exitCode=0
Oct 30 01:13:40.025607 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:40.025488    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf" event={"ID":"d7a01fa7-260e-466c-b416-aad24c892577","Type":"ContainerDied","Data":"45fd3fd7fd48861b334246daf47e6084e429dd675eb5c3f8ecc5466a4e3259e8"}
Oct 30 01:13:40.025607 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:40.025529    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf" event={"ID":"d7a01fa7-260e-466c-b416-aad24c892577","Type":"ContainerDied","Data":"82559695e39d4a80eca4d19d9ea11d76d3d0bf8ea97bc3a7e7fcca91b8424be3"}
Oct 30 01:13:40.025607 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:40.025541    4393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82559695e39d4a80eca4d19d9ea11d76d3d0bf8ea97bc3a7e7fcca91b8424be3"
Oct 30 01:13:40.038903 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:40.038870    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf"
Oct 30 01:13:40.152746 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:40.152709    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7a01fa7-260e-466c-b416-aad24c892577-kube-api-access-ztxrv\" (UniqueName: \"kubernetes.io/projected/d7a01fa7-260e-466c-b416-aad24c892577-kube-api-access-ztxrv\") pod \"d7a01fa7-260e-466c-b416-aad24c892577\" (UID: \"d7a01fa7-260e-466c-b416-aad24c892577\") "
Oct 30 01:13:40.152971 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:40.152846    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7a01fa7-260e-466c-b416-aad24c892577-home\" (UniqueName: \"kubernetes.io/empty-dir/d7a01fa7-260e-466c-b416-aad24c892577-home\") pod \"d7a01fa7-260e-466c-b416-aad24c892577\" (UID: \"d7a01fa7-260e-466c-b416-aad24c892577\") "
Oct 30 01:13:40.152971 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:40.152906    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7a01fa7-260e-466c-b416-aad24c892577-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7a01fa7-260e-466c-b416-aad24c892577-tls-certs\") pod \"d7a01fa7-260e-466c-b416-aad24c892577\" (UID: \"d7a01fa7-260e-466c-b416-aad24c892577\") "
Oct 30 01:13:40.152971 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:40.152925    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7a01fa7-260e-466c-b416-aad24c892577-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7a01fa7-260e-466c-b416-aad24c892577-tmp-dir\") pod \"d7a01fa7-260e-466c-b416-aad24c892577\" (UID: \"d7a01fa7-260e-466c-b416-aad24c892577\") "
Oct 30 01:13:40.153145 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:40.153005    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7a01fa7-260e-466c-b416-aad24c892577-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7a01fa7-260e-466c-b416-aad24c892577-model-cache\") pod \"d7a01fa7-260e-466c-b416-aad24c892577\" (UID: \"d7a01fa7-260e-466c-b416-aad24c892577\") "
Oct 30 01:13:40.153145 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:40.153072    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7a01fa7-260e-466c-b416-aad24c892577-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7a01fa7-260e-466c-b416-aad24c892577-dshm\") pod \"d7a01fa7-260e-466c-b416-aad24c892577\" (UID: \"d7a01fa7-260e-466c-b416-aad24c892577\") "
Oct 30 01:13:40.153258 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:40.153195    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7a01fa7-260e-466c-b416-aad24c892577-tmp-dir" pod "d7a01fa7-260e-466c-b416-aad24c892577" (UID: "d7a01fa7-260e-466c-b416-aad24c892577"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:40.153258 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:40.153213    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7a01fa7-260e-466c-b416-aad24c892577-home" pod "d7a01fa7-260e-466c-b416-aad24c892577" (UID: "d7a01fa7-260e-466c-b416-aad24c892577"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:40.153412 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:40.153363    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7a01fa7-260e-466c-b416-aad24c892577-model-cache" pod "d7a01fa7-260e-466c-b416-aad24c892577" (UID: "d7a01fa7-260e-466c-b416-aad24c892577"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:40.153824 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:40.153799    4393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7a01fa7-260e-466c-b416-aad24c892577-tmp-dir\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:13:40.153824 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:40.153825    4393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7a01fa7-260e-466c-b416-aad24c892577-model-cache\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:13:40.154001 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:40.153838    4393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7a01fa7-260e-466c-b416-aad24c892577-home\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:13:40.155638 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:40.155608    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7a01fa7-260e-466c-b416-aad24c892577-kube-api-access-ztxrv" pod "d7a01fa7-260e-466c-b416-aad24c892577" (UID: "d7a01fa7-260e-466c-b416-aad24c892577"). InnerVolumeSpecName "kube-api-access-ztxrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:13:40.155915 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:40.155890    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7a01fa7-260e-466c-b416-aad24c892577-tls-certs" pod "d7a01fa7-260e-466c-b416-aad24c892577" (UID: "d7a01fa7-260e-466c-b416-aad24c892577"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:13:40.156183 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:40.156146    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7a01fa7-260e-466c-b416-aad24c892577-dshm" pod "d7a01fa7-260e-466c-b416-aad24c892577" (UID: "d7a01fa7-260e-466c-b416-aad24c892577"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:13:40.255187 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:40.255100    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztxrv\" (UniqueName: \"kubernetes.io/projected/d7a01fa7-260e-466c-b416-aad24c892577-kube-api-access-ztxrv\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:13:40.255187 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:40.255129    4393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7a01fa7-260e-466c-b416-aad24c892577-tls-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:13:40.255187 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:40.255141    4393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7a01fa7-260e-466c-b416-aad24c892577-dshm\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:13:40.976116 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:40.976053    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf" podUID="d7a01fa7-260e-466c-b416-aad24c892577" containerName="main" probeResult="failure" output="Get \"https://10.133.0.169:8000/ready\": context deadline exceeded"
Oct 30 01:13:41.034252 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:41.034205    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf"
Oct 30 01:13:41.088494 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:41.088463    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf"]
Oct 30 01:13:41.098110 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:41.098085    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-93079c/auth-test-model-93079c-kserve-775c79c645-55zgf"]
Oct 30 01:13:41.727358 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:41.727328    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7a01fa7-260e-466c-b416-aad24c892577" path="/var/lib/kubelet/pods/d7a01fa7-260e-466c-b416-aad24c892577/volumes"
Oct 30 01:13:45.055594 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.055560    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"]
Oct 30 01:13:45.056621 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.056597    4393 cpu_manager.go:414] "containerMap: removing container" podUID="ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4" containerName="main"
Oct 30 01:13:45.056621 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.056622    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4" containerName="main"
Oct 30 01:13:45.056817 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.056691    4393 cpu_manager.go:414] "containerMap: removing container" podUID="d7a01fa7-260e-466c-b416-aad24c892577" containerName="main"
Oct 30 01:13:45.056817 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.056700    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7a01fa7-260e-466c-b416-aad24c892577" containerName="main"
Oct 30 01:13:45.056973 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.056956    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca5cbcaf-46cc-4a51-b372-10a6fd84f2f4" containerName="main"
Oct 30 01:13:45.057035 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.056992    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7a01fa7-260e-466c-b416-aad24c892577" containerName="main"
Oct 30 01:13:45.062588 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.062545    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"
Oct 30 01:13:45.086366 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.086261    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"]
Oct 30 01:13:45.090032 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.090000    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-009e78-openshift-default-dockercfg-c68np\""
Oct 30 01:13:45.165417 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.165381    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e34134b-9305-4376-b55b-c08c72cfbedd-credential-socket\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp\" (UID: \"1e34134b-9305-4376-b55b-c08c72cfbedd\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"
Oct 30 01:13:45.165635 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.165433    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e34134b-9305-4376-b55b-c08c72cfbedd-istiod-ca-cert\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp\" (UID: \"1e34134b-9305-4376-b55b-c08c72cfbedd\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"
Oct 30 01:13:45.165635 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.165466    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e34134b-9305-4376-b55b-c08c72cfbedd-istio-data\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp\" (UID: \"1e34134b-9305-4376-b55b-c08c72cfbedd\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"
Oct 30 01:13:45.166030 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.165799    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e34134b-9305-4376-b55b-c08c72cfbedd-workload-certs\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp\" (UID: \"1e34134b-9305-4376-b55b-c08c72cfbedd\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"
Oct 30 01:13:45.166030 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.165830    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e34134b-9305-4376-b55b-c08c72cfbedd-istio-token\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp\" (UID: \"1e34134b-9305-4376-b55b-c08c72cfbedd\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"
Oct 30 01:13:45.166030 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.165847    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e34134b-9305-4376-b55b-c08c72cfbedd-istio-podinfo\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp\" (UID: \"1e34134b-9305-4376-b55b-c08c72cfbedd\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"
Oct 30 01:13:45.166030 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.165941    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e34134b-9305-4376-b55b-c08c72cfbedd-workload-socket\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp\" (UID: \"1e34134b-9305-4376-b55b-c08c72cfbedd\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"
Oct 30 01:13:45.166030 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.165985    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e34134b-9305-4376-b55b-c08c72cfbedd-istio-envoy\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp\" (UID: \"1e34134b-9305-4376-b55b-c08c72cfbedd\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"
Oct 30 01:13:45.166030 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.166009    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r42zf\" (UniqueName: \"kubernetes.io/projected/1e34134b-9305-4376-b55b-c08c72cfbedd-kube-api-access-r42zf\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp\" (UID: \"1e34134b-9305-4376-b55b-c08c72cfbedd\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"
Oct 30 01:13:45.267486 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.267450    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e34134b-9305-4376-b55b-c08c72cfbedd-credential-socket\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp\" (UID: \"1e34134b-9305-4376-b55b-c08c72cfbedd\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"
Oct 30 01:13:45.267658 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.267492    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e34134b-9305-4376-b55b-c08c72cfbedd-istiod-ca-cert\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp\" (UID: \"1e34134b-9305-4376-b55b-c08c72cfbedd\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"
Oct 30 01:13:45.267658 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.267548    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e34134b-9305-4376-b55b-c08c72cfbedd-istio-data\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp\" (UID: \"1e34134b-9305-4376-b55b-c08c72cfbedd\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"
Oct 30 01:13:45.267771 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.267665    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e34134b-9305-4376-b55b-c08c72cfbedd-workload-certs\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp\" (UID: \"1e34134b-9305-4376-b55b-c08c72cfbedd\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"
Oct 30 01:13:45.267771 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.267690    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e34134b-9305-4376-b55b-c08c72cfbedd-istio-token\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp\" (UID: \"1e34134b-9305-4376-b55b-c08c72cfbedd\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"
Oct 30 01:13:45.267771 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.267718    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e34134b-9305-4376-b55b-c08c72cfbedd-istio-podinfo\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp\" (UID: \"1e34134b-9305-4376-b55b-c08c72cfbedd\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"
Oct 30 01:13:45.267968 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.267902    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e34134b-9305-4376-b55b-c08c72cfbedd-workload-socket\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp\" (UID: \"1e34134b-9305-4376-b55b-c08c72cfbedd\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"
Oct 30 01:13:45.267968 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.267960    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e34134b-9305-4376-b55b-c08c72cfbedd-istio-envoy\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp\" (UID: \"1e34134b-9305-4376-b55b-c08c72cfbedd\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"
Oct 30 01:13:45.268103 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.267983    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e34134b-9305-4376-b55b-c08c72cfbedd-credential-socket\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp\" (UID: \"1e34134b-9305-4376-b55b-c08c72cfbedd\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"
Oct 30 01:13:45.268103 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.267984    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r42zf\" (UniqueName: \"kubernetes.io/projected/1e34134b-9305-4376-b55b-c08c72cfbedd-kube-api-access-r42zf\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp\" (UID: \"1e34134b-9305-4376-b55b-c08c72cfbedd\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"
Oct 30 01:13:45.268236 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.268217    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e34134b-9305-4376-b55b-c08c72cfbedd-workload-socket\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp\" (UID: \"1e34134b-9305-4376-b55b-c08c72cfbedd\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"
Oct 30 01:13:45.268454 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.268434    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e34134b-9305-4376-b55b-c08c72cfbedd-workload-certs\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp\" (UID: \"1e34134b-9305-4376-b55b-c08c72cfbedd\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"
Oct 30 01:13:45.268541 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.268436    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e34134b-9305-4376-b55b-c08c72cfbedd-istiod-ca-cert\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp\" (UID: \"1e34134b-9305-4376-b55b-c08c72cfbedd\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"
Oct 30 01:13:45.268541 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.268530    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e34134b-9305-4376-b55b-c08c72cfbedd-istio-data\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp\" (UID: \"1e34134b-9305-4376-b55b-c08c72cfbedd\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"
Oct 30 01:13:45.270634 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.270606    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e34134b-9305-4376-b55b-c08c72cfbedd-istio-envoy\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp\" (UID: \"1e34134b-9305-4376-b55b-c08c72cfbedd\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"
Oct 30 01:13:45.270751 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.270678    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e34134b-9305-4376-b55b-c08c72cfbedd-istio-podinfo\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp\" (UID: \"1e34134b-9305-4376-b55b-c08c72cfbedd\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"
Oct 30 01:13:45.291689 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.291656    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e34134b-9305-4376-b55b-c08c72cfbedd-istio-token\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp\" (UID: \"1e34134b-9305-4376-b55b-c08c72cfbedd\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"
Oct 30 01:13:45.291908 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.291748    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r42zf\" (UniqueName: \"kubernetes.io/projected/1e34134b-9305-4376-b55b-c08c72cfbedd-kube-api-access-r42zf\") pod \"e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp\" (UID: \"1e34134b-9305-4376-b55b-c08c72cfbedd\") " pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"
Oct 30 01:13:45.387998 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.387963    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"
Oct 30 01:13:45.760857 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:13:45.760815    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e34134b_9305_4376_b55b_c08c72cfbedd.slice/crio-f309f35955db877fca413297bd6fc31b6d6d056923dd48ff0c9e2e92f5ed8c5f WatchSource:0}: Error finding container f309f35955db877fca413297bd6fc31b6d6d056923dd48ff0c9e2e92f5ed8c5f: Status 404 returned error can't find the container with id f309f35955db877fca413297bd6fc31b6d6d056923dd48ff0c9e2e92f5ed8c5f
Oct 30 01:13:45.761947 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.761915    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"]
Oct 30 01:13:45.763980 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.763944    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:13:45.764111 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.764024    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:13:45.764111 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:45.764070    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:13:46.079669 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:46.079576    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp" event={"ID":"1e34134b-9305-4376-b55b-c08c72cfbedd","Type":"ContainerStarted","Data":"885834e85744070f20a70adb7d0bf6a74fd431d662c798875de1b6be267e6bc3"}
Oct 30 01:13:46.079669 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:46.079635    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp" event={"ID":"1e34134b-9305-4376-b55b-c08c72cfbedd","Type":"ContainerStarted","Data":"f309f35955db877fca413297bd6fc31b6d6d056923dd48ff0c9e2e92f5ed8c5f"}
Oct 30 01:13:46.127522 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:46.127466    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp" podStartSLOduration=1.12745292 podStartE2EDuration="1.12745292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:46.124713587 +0000 UTC m=+2516.512389975" watchObservedRunningTime="2026-10-30 01:13:46.12745292 +0000 UTC m=+2516.515129564"
Oct 30 01:13:46.388526 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:46.388486    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"
Oct 30 01:13:46.390294 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:46.390270    4393 patch_prober.go:29] interesting pod/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 30 01:13:46.390413 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:46.390326    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp" podUID="1e34134b-9305-4376-b55b-c08c72cfbedd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 30 01:13:47.388938 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:47.388901    4393 patch_prober.go:29] interesting pod/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 30 01:13:47.389350 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:47.388953    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp" podUID="1e34134b-9305-4376-b55b-c08c72cfbedd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 30 01:13:48.388363 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:48.388326    4393 patch_prober.go:29] interesting pod/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 30 01:13:48.388547 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:48.388391    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp" podUID="1e34134b-9305-4376-b55b-c08c72cfbedd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 30 01:13:49.389235 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:49.389196    4393 patch_prober.go:29] interesting pod/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 30 01:13:49.389665 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:49.389257    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp" podUID="1e34134b-9305-4376-b55b-c08c72cfbedd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 30 01:13:50.388518 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:50.388487    4393 patch_prober.go:29] interesting pod/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 30 01:13:50.388755 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:50.388735    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp" podUID="1e34134b-9305-4376-b55b-c08c72cfbedd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 30 01:13:51.393807 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:51.392817    4393 patch_prober.go:29] interesting pod/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 30 01:13:51.393807 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:51.392878    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp" podUID="1e34134b-9305-4376-b55b-c08c72cfbedd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 30 01:13:52.389333 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:52.389288    4393 patch_prober.go:29] interesting pod/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Oct 30 01:13:52.389564 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:52.389375    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp" podUID="1e34134b-9305-4376-b55b-c08c72cfbedd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Oct 30 01:13:53.394092 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:53.394059    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"
Oct 30 01:13:53.394626 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:53.394440    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"
Oct 30 01:13:53.395216 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:53.395199    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"
Oct 30 01:13:55.185521 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:55.185489    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8816-b56c8b"]
Oct 30 01:13:55.190853 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:55.190830    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8816-b56c8b"
Oct 30 01:13:55.206693 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:55.206667    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8816-b56c8b"]
Oct 30 01:13:55.212099 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:55.212073    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nrxkf\""
Oct 30 01:13:55.213642 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:55.213511    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7t72\" (UniqueName: \"kubernetes.io/projected/bd0793f7-b4b5-4539-a837-030e173e5297-kube-api-access-s7t72\") pod \"test-curl-8816-b56c8b\" (UID: \"bd0793f7-b4b5-4539-a837-030e173e5297\") " pod="opendatahub/test-curl-8816-b56c8b"
Oct 30 01:13:55.314557 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:55.314514    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7t72\" (UniqueName: \"kubernetes.io/projected/bd0793f7-b4b5-4539-a837-030e173e5297-kube-api-access-s7t72\") pod \"test-curl-8816-b56c8b\" (UID: \"bd0793f7-b4b5-4539-a837-030e173e5297\") " pod="opendatahub/test-curl-8816-b56c8b"
Oct 30 01:13:55.337363 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:55.337332    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7t72\" (UniqueName: \"kubernetes.io/projected/bd0793f7-b4b5-4539-a837-030e173e5297-kube-api-access-s7t72\") pod \"test-curl-8816-b56c8b\" (UID: \"bd0793f7-b4b5-4539-a837-030e173e5297\") " pod="opendatahub/test-curl-8816-b56c8b"
Oct 30 01:13:55.506147 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:55.506067    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8816-b56c8b"
Oct 30 01:13:55.822194 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:55.822164    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8816-b56c8b"]
Oct 30 01:13:56.201962 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:56.201921    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8816-b56c8b" event={"ID":"bd0793f7-b4b5-4539-a837-030e173e5297","Type":"ContainerStarted","Data":"f11f4aac463ad9999d42fc298b1914b0de9d46c966b50942814e3bff0ce2ad7b"}
Oct 30 01:13:56.201962 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:56.201973    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8816-b56c8b" event={"ID":"bd0793f7-b4b5-4539-a837-030e173e5297","Type":"ContainerStarted","Data":"19f8c4d735b783f56d032c3c196bbad854a0498c85452d39478a4e84b6a0d8f6"}
Oct 30 01:13:56.239438 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:56.239389    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8816-b56c8b" podStartSLOduration=1.239375556 podStartE2EDuration="1.239375556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:56.237981925 +0000 UTC m=+2526.625658314" watchObservedRunningTime="2026-10-30 01:13:56.239375556 +0000 UTC m=+2526.627051944"
Oct 30 01:13:56.513365 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:13:56.513274    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:13:57.180660 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:57.180623    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8816-b56c8b"]
Oct 30 01:13:57.188835 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:57.182944    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8816-b56c8b"]
Oct 30 01:13:57.331168 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:57.331135    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8816-45db46"]
Oct 30 01:13:57.332099 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:57.332081    4393 cpu_manager.go:414] "containerMap: removing container" podUID="bd0793f7-b4b5-4539-a837-030e173e5297" containerName="test-curl-8816-b56c8b"
Oct 30 01:13:57.332151 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:57.332100    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd0793f7-b4b5-4539-a837-030e173e5297" containerName="test-curl-8816-b56c8b"
Oct 30 01:13:57.332248 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:57.332238    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd0793f7-b4b5-4539-a837-030e173e5297" containerName="test-curl-8816-b56c8b"
Oct 30 01:13:57.336430 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:57.336410    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8816-45db46"
Oct 30 01:13:57.352945 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:57.352918    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8816-45db46"]
Oct 30 01:13:57.444974 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:57.444885    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr6xg\" (UniqueName: \"kubernetes.io/projected/361d0ab2-b911-4adf-9307-e295d8d26b47-kube-api-access-cr6xg\") pod \"test-curl-8816-45db46\" (UID: \"361d0ab2-b911-4adf-9307-e295d8d26b47\") " pod="opendatahub/test-curl-8816-45db46"
Oct 30 01:13:57.546153 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:57.546123    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr6xg\" (UniqueName: \"kubernetes.io/projected/361d0ab2-b911-4adf-9307-e295d8d26b47-kube-api-access-cr6xg\") pod \"test-curl-8816-45db46\" (UID: \"361d0ab2-b911-4adf-9307-e295d8d26b47\") " pod="opendatahub/test-curl-8816-45db46"
Oct 30 01:13:57.568349 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:57.568313    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr6xg\" (UniqueName: \"kubernetes.io/projected/361d0ab2-b911-4adf-9307-e295d8d26b47-kube-api-access-cr6xg\") pod \"test-curl-8816-45db46\" (UID: \"361d0ab2-b911-4adf-9307-e295d8d26b47\") " pod="opendatahub/test-curl-8816-45db46"
Oct 30 01:13:57.651235 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:57.651206    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8816-45db46"
Oct 30 01:13:57.967371 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:13:57.967332    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod361d0ab2_b911_4adf_9307_e295d8d26b47.slice/crio-ab0e8318ad6042a16b39c9856f2643c3a083de56b00ae5875f3ffc327614baa0 WatchSource:0}: Error finding container ab0e8318ad6042a16b39c9856f2643c3a083de56b00ae5875f3ffc327614baa0: Status 404 returned error can't find the container with id ab0e8318ad6042a16b39c9856f2643c3a083de56b00ae5875f3ffc327614baa0
Oct 30 01:13:57.968359 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:57.968301    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8816-45db46"]
Oct 30 01:13:58.219592 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:58.219495    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8816-45db46" event={"ID":"361d0ab2-b911-4adf-9307-e295d8d26b47","Type":"ContainerStarted","Data":"8e4f954f731460be4adfa242b48fb2df762a3d9ce0fdd01af1a2bbf7048d21da"}
Oct 30 01:13:58.219592 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:58.219546    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8816-45db46" event={"ID":"361d0ab2-b911-4adf-9307-e295d8d26b47","Type":"ContainerStarted","Data":"ab0e8318ad6042a16b39c9856f2643c3a083de56b00ae5875f3ffc327614baa0"}
Oct 30 01:13:58.219592 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:58.219541    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8816-b56c8b" podUID="bd0793f7-b4b5-4539-a837-030e173e5297" containerName="test-curl-8816-b56c8b" containerID="cri-o://f11f4aac463ad9999d42fc298b1914b0de9d46c966b50942814e3bff0ce2ad7b" gracePeriod=2
Oct 30 01:13:58.260195 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:58.260143    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8816-45db46" podStartSLOduration=1.260130336 podStartE2EDuration="1.260130336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:13:58.256236843 +0000 UTC m=+2528.643913233" watchObservedRunningTime="2026-10-30 01:13:58.260130336 +0000 UTC m=+2528.647806725"
Oct 30 01:13:59.177250 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:59.177213    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8816-45db46"]
Oct 30 01:13:59.182896 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:13:59.182862    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8816-45db46"]
Oct 30 01:14:00.249814 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:00.247115    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8816-45db46" podUID="361d0ab2-b911-4adf-9307-e295d8d26b47" containerName="test-curl-8816-45db46" containerID="cri-o://8e4f954f731460be4adfa242b48fb2df762a3d9ce0fdd01af1a2bbf7048d21da" gracePeriod=2
Oct 30 01:14:00.460877 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:00.460850    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8816-b56c8b"
Oct 30 01:14:00.492987 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:00.492914    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd0793f7-b4b5-4539-a837-030e173e5297-kube-api-access-s7t72\" (UniqueName: \"kubernetes.io/projected/bd0793f7-b4b5-4539-a837-030e173e5297-kube-api-access-s7t72\") pod \"bd0793f7-b4b5-4539-a837-030e173e5297\" (UID: \"bd0793f7-b4b5-4539-a837-030e173e5297\") "
Oct 30 01:14:00.495346 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:00.495323    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd0793f7-b4b5-4539-a837-030e173e5297-kube-api-access-s7t72" pod "bd0793f7-b4b5-4539-a837-030e173e5297" (UID: "bd0793f7-b4b5-4539-a837-030e173e5297"). InnerVolumeSpecName "kube-api-access-s7t72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:00.595397 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:00.595364    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7t72\" (UniqueName: \"kubernetes.io/projected/bd0793f7-b4b5-4539-a837-030e173e5297-kube-api-access-s7t72\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:14:01.256814 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:01.256762    4393 generic.go:356] "Generic (PLEG): container finished" podID="bd0793f7-b4b5-4539-a837-030e173e5297" containerID="f11f4aac463ad9999d42fc298b1914b0de9d46c966b50942814e3bff0ce2ad7b" exitCode=137
Oct 30 01:14:01.257282 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:01.256822    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8816-b56c8b"
Oct 30 01:14:01.257282 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:01.256846    4393 scope.go:122] "RemoveContainer" containerID="f11f4aac463ad9999d42fc298b1914b0de9d46c966b50942814e3bff0ce2ad7b"
Oct 30 01:14:01.268399 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:01.268379    4393 scope.go:122] "RemoveContainer" containerID="f11f4aac463ad9999d42fc298b1914b0de9d46c966b50942814e3bff0ce2ad7b"
Oct 30 01:14:01.268676 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:14:01.268660    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f11f4aac463ad9999d42fc298b1914b0de9d46c966b50942814e3bff0ce2ad7b\": container with ID starting with f11f4aac463ad9999d42fc298b1914b0de9d46c966b50942814e3bff0ce2ad7b not found: ID does not exist" containerID="f11f4aac463ad9999d42fc298b1914b0de9d46c966b50942814e3bff0ce2ad7b"
Oct 30 01:14:01.268715 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:01.268687    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f11f4aac463ad9999d42fc298b1914b0de9d46c966b50942814e3bff0ce2ad7b"} err="failed to get container status \"f11f4aac463ad9999d42fc298b1914b0de9d46c966b50942814e3bff0ce2ad7b\": rpc error: code = NotFound desc = could not find container \"f11f4aac463ad9999d42fc298b1914b0de9d46c966b50942814e3bff0ce2ad7b\": container with ID starting with f11f4aac463ad9999d42fc298b1914b0de9d46c966b50942814e3bff0ce2ad7b not found: ID does not exist"
Oct 30 01:14:01.279155 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:14:01.279128    4393 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8816-b56c8b\" is forbidden: User \"system:node:ip-10-0-135-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-56.ec2.internal' and this object" podUID="bd0793f7-b4b5-4539-a837-030e173e5297" pod="opendatahub/test-curl-8816-b56c8b"
Oct 30 01:14:01.726919 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:01.726887    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd0793f7-b4b5-4539-a837-030e173e5297" path="/var/lib/kubelet/pods/bd0793f7-b4b5-4539-a837-030e173e5297/volumes"
Oct 30 01:14:02.478909 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:02.478880    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8816-45db46"
Oct 30 01:14:02.523148 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:02.523076    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/361d0ab2-b911-4adf-9307-e295d8d26b47-kube-api-access-cr6xg\" (UniqueName: \"kubernetes.io/projected/361d0ab2-b911-4adf-9307-e295d8d26b47-kube-api-access-cr6xg\") pod \"361d0ab2-b911-4adf-9307-e295d8d26b47\" (UID: \"361d0ab2-b911-4adf-9307-e295d8d26b47\") "
Oct 30 01:14:02.525290 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:02.525266    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/361d0ab2-b911-4adf-9307-e295d8d26b47-kube-api-access-cr6xg" pod "361d0ab2-b911-4adf-9307-e295d8d26b47" (UID: "361d0ab2-b911-4adf-9307-e295d8d26b47"). InnerVolumeSpecName "kube-api-access-cr6xg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:02.625588 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:02.625548    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr6xg\" (UniqueName: \"kubernetes.io/projected/361d0ab2-b911-4adf-9307-e295d8d26b47-kube-api-access-cr6xg\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:14:03.292162 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:03.292127    4393 generic.go:356] "Generic (PLEG): container finished" podID="361d0ab2-b911-4adf-9307-e295d8d26b47" containerID="8e4f954f731460be4adfa242b48fb2df762a3d9ce0fdd01af1a2bbf7048d21da" exitCode=137
Oct 30 01:14:03.292697 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:03.292431    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8816-45db46"
Oct 30 01:14:03.293940 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:03.292473    4393 scope.go:122] "RemoveContainer" containerID="8e4f954f731460be4adfa242b48fb2df762a3d9ce0fdd01af1a2bbf7048d21da"
Oct 30 01:14:03.338061 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:14:03.332364    4393 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8816-45db46\" is forbidden: User \"system:node:ip-10-0-135-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-56.ec2.internal' and this object" podUID="361d0ab2-b911-4adf-9307-e295d8d26b47" pod="opendatahub/test-curl-8816-45db46"
Oct 30 01:14:03.436339 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:03.436300    4393 scope.go:122] "RemoveContainer" containerID="8e4f954f731460be4adfa242b48fb2df762a3d9ce0fdd01af1a2bbf7048d21da"
Oct 30 01:14:03.437078 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:14:03.437034    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e4f954f731460be4adfa242b48fb2df762a3d9ce0fdd01af1a2bbf7048d21da\": container with ID starting with 8e4f954f731460be4adfa242b48fb2df762a3d9ce0fdd01af1a2bbf7048d21da not found: ID does not exist" containerID="8e4f954f731460be4adfa242b48fb2df762a3d9ce0fdd01af1a2bbf7048d21da"
Oct 30 01:14:03.437282 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:03.437249    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e4f954f731460be4adfa242b48fb2df762a3d9ce0fdd01af1a2bbf7048d21da"} err="failed to get container status \"8e4f954f731460be4adfa242b48fb2df762a3d9ce0fdd01af1a2bbf7048d21da\": rpc error: code = NotFound desc = could not find container \"8e4f954f731460be4adfa242b48fb2df762a3d9ce0fdd01af1a2bbf7048d21da\": container with ID starting with 8e4f954f731460be4adfa242b48fb2df762a3d9ce0fdd01af1a2bbf7048d21da not found: ID does not exist"
Oct 30 01:14:03.727314 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:03.727277    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="361d0ab2-b911-4adf-9307-e295d8d26b47" path="/var/lib/kubelet/pods/361d0ab2-b911-4adf-9307-e295d8d26b47/volumes"
Oct 30 01:14:26.113474 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:26.113441    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z"]
Oct 30 01:14:26.114457 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:26.114437    4393 cpu_manager.go:414] "containerMap: removing container" podUID="361d0ab2-b911-4adf-9307-e295d8d26b47" containerName="test-curl-8816-45db46"
Oct 30 01:14:26.114516 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:26.114459    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="361d0ab2-b911-4adf-9307-e295d8d26b47" containerName="test-curl-8816-45db46"
Oct 30 01:14:26.114610 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:26.114600    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="361d0ab2-b911-4adf-9307-e295d8d26b47" containerName="test-curl-8816-45db46"
Oct 30 01:14:26.120430 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:26.119857    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z"
Oct 30 01:14:26.139685 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:26.139654    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z"]
Oct 30 01:14:26.144962 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:26.144922    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-01215f-serving-cert\""
Oct 30 01:14:26.217440 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:26.217403    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/914a8a1b-f13f-4a16-be9d-ebcbda26ac98-maas-api-tls\") pod \"maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z\" (UID: \"914a8a1b-f13f-4a16-be9d-ebcbda26ac98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z"
Oct 30 01:14:26.217629 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:26.217559    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/914a8a1b-f13f-4a16-be9d-ebcbda26ac98-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z\" (UID: \"914a8a1b-f13f-4a16-be9d-ebcbda26ac98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z"
Oct 30 01:14:26.217629 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:26.217604    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz8qg\" (UniqueName: \"kubernetes.io/projected/914a8a1b-f13f-4a16-be9d-ebcbda26ac98-kube-api-access-sz8qg\") pod \"maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z\" (UID: \"914a8a1b-f13f-4a16-be9d-ebcbda26ac98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z"
Oct 30 01:14:26.217764 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:26.217750    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/914a8a1b-f13f-4a16-be9d-ebcbda26ac98-metrics-certs\") pod \"maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z\" (UID: \"914a8a1b-f13f-4a16-be9d-ebcbda26ac98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z"
Oct 30 01:14:26.318885 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:26.318848    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/914a8a1b-f13f-4a16-be9d-ebcbda26ac98-metrics-certs\") pod \"maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z\" (UID: \"914a8a1b-f13f-4a16-be9d-ebcbda26ac98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z"
Oct 30 01:14:26.319050 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:26.319008    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/914a8a1b-f13f-4a16-be9d-ebcbda26ac98-maas-api-tls\") pod \"maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z\" (UID: \"914a8a1b-f13f-4a16-be9d-ebcbda26ac98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z"
Oct 30 01:14:26.319118 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:26.319057    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/914a8a1b-f13f-4a16-be9d-ebcbda26ac98-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z\" (UID: \"914a8a1b-f13f-4a16-be9d-ebcbda26ac98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z"
Oct 30 01:14:26.319118 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:26.319079    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz8qg\" (UniqueName: \"kubernetes.io/projected/914a8a1b-f13f-4a16-be9d-ebcbda26ac98-kube-api-access-sz8qg\") pod \"maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z\" (UID: \"914a8a1b-f13f-4a16-be9d-ebcbda26ac98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z"
Oct 30 01:14:26.319707 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:26.319681    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/914a8a1b-f13f-4a16-be9d-ebcbda26ac98-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z\" (UID: \"914a8a1b-f13f-4a16-be9d-ebcbda26ac98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z"
Oct 30 01:14:26.321558 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:26.321538    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/914a8a1b-f13f-4a16-be9d-ebcbda26ac98-metrics-certs\") pod \"maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z\" (UID: \"914a8a1b-f13f-4a16-be9d-ebcbda26ac98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z"
Oct 30 01:14:26.321628 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:26.321586    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/914a8a1b-f13f-4a16-be9d-ebcbda26ac98-maas-api-tls\") pod \"maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z\" (UID: \"914a8a1b-f13f-4a16-be9d-ebcbda26ac98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z"
Oct 30 01:14:26.341656 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:26.341632    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz8qg\" (UniqueName: \"kubernetes.io/projected/914a8a1b-f13f-4a16-be9d-ebcbda26ac98-kube-api-access-sz8qg\") pod \"maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z\" (UID: \"914a8a1b-f13f-4a16-be9d-ebcbda26ac98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z"
Oct 30 01:14:26.447174 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:26.447136    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z"
Oct 30 01:14:26.520656 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:14:26.520584    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:14:26.775768 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:14:26.775740    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod914a8a1b_f13f_4a16_be9d_ebcbda26ac98.slice/crio-8389ad2072eaabefae4f1d2edd5fb06f38850228cb3d60d1e7c367c36e9df209 WatchSource:0}: Error finding container 8389ad2072eaabefae4f1d2edd5fb06f38850228cb3d60d1e7c367c36e9df209: Status 404 returned error can't find the container with id 8389ad2072eaabefae4f1d2edd5fb06f38850228cb3d60d1e7c367c36e9df209
Oct 30 01:14:26.775944 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:26.775844    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z"]
Oct 30 01:14:27.557812 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:27.554116    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z" event={"ID":"914a8a1b-f13f-4a16-be9d-ebcbda26ac98","Type":"ContainerStarted","Data":"a5db9bdc0ca748998b4676d0fe7822e931a5372c96b50b7ece4973c2d2e2af44"}
Oct 30 01:14:27.557812 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:27.554201    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z" event={"ID":"914a8a1b-f13f-4a16-be9d-ebcbda26ac98","Type":"ContainerStarted","Data":"8389ad2072eaabefae4f1d2edd5fb06f38850228cb3d60d1e7c367c36e9df209"}
Oct 30 01:14:27.557812 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:27.555127    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z"
Oct 30 01:14:27.610729 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:27.610678    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z" podStartSLOduration=1.290555963 podStartE2EDuration="1.610664564s" totalImagesPullingTime="320.108601ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:14:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:14:27.604525725 +0000 UTC m=+2557.992202113" watchObservedRunningTime="2026-10-30 01:14:27.610664564 +0000 UTC m=+2557.998340953"
Oct 30 01:14:33.569460 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:33.569379    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z"
Oct 30 01:14:35.062358 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.062313    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb"]
Oct 30 01:14:35.062833 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.062622    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb" podUID="2c3ad599-c376-47ab-bbf0-fe6c859d6c9c" containerName="maas-api" containerID="cri-o://0f31035d565898cfe4448ce2da149ab4d1959d4c1ba6916bbfccff7a98f021c3" gracePeriod=30
Oct 30 01:14:35.264627 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.264571    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-68746cfdb7-hsxp2"]
Oct 30 01:14:35.265537 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.265493    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-68746cfdb7-hsxp2" podUID="8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a" containerName="payload-pre-processing" containerID="cri-o://8e2c4f087ae3925b97cf39a9a161436fda2755954ebc485e501479d19a4f660e" gracePeriod=30
Oct 30 01:14:35.451500 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.451472    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb"
Oct 30 01:14:35.501171 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.499431    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4"]
Oct 30 01:14:35.501171 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.499668    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4" podUID="734513f7-51fc-49f0-98a4-bf801eb09fb1" containerName="maas-api" containerID="cri-o://7097715569e230b160ade9a20ca375c64c8ec117f1cd65d9832406ca8c125d46" gracePeriod=30
Oct 30 01:14:35.595168 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.595141    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c3ad599-c376-47ab-bbf0-fe6c859d6c9c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c3ad599-c376-47ab-bbf0-fe6c859d6c9c-metrics-certs\") pod \"2c3ad599-c376-47ab-bbf0-fe6c859d6c9c\" (UID: \"2c3ad599-c376-47ab-bbf0-fe6c859d6c9c\") "
Oct 30 01:14:35.595351 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.595296    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c3ad599-c376-47ab-bbf0-fe6c859d6c9c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c3ad599-c376-47ab-bbf0-fe6c859d6c9c-openshift-service-ca\") pod \"2c3ad599-c376-47ab-bbf0-fe6c859d6c9c\" (UID: \"2c3ad599-c376-47ab-bbf0-fe6c859d6c9c\") "
Oct 30 01:14:35.595413 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.595346    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c3ad599-c376-47ab-bbf0-fe6c859d6c9c-kube-api-access-9gk2j\" (UniqueName: \"kubernetes.io/projected/2c3ad599-c376-47ab-bbf0-fe6c859d6c9c-kube-api-access-9gk2j\") pod \"2c3ad599-c376-47ab-bbf0-fe6c859d6c9c\" (UID: \"2c3ad599-c376-47ab-bbf0-fe6c859d6c9c\") "
Oct 30 01:14:35.595479 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.595461    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c3ad599-c376-47ab-bbf0-fe6c859d6c9c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c3ad599-c376-47ab-bbf0-fe6c859d6c9c-maas-api-tls\") pod \"2c3ad599-c376-47ab-bbf0-fe6c859d6c9c\" (UID: \"2c3ad599-c376-47ab-bbf0-fe6c859d6c9c\") "
Oct 30 01:14:35.595870 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.595772    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c3ad599-c376-47ab-bbf0-fe6c859d6c9c-openshift-service-ca" pod "2c3ad599-c376-47ab-bbf0-fe6c859d6c9c" (UID: "2c3ad599-c376-47ab-bbf0-fe6c859d6c9c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:14:35.596137 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.596113    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c3ad599-c376-47ab-bbf0-fe6c859d6c9c-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:14:35.597901 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.597862    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c3ad599-c376-47ab-bbf0-fe6c859d6c9c-metrics-certs" pod "2c3ad599-c376-47ab-bbf0-fe6c859d6c9c" (UID: "2c3ad599-c376-47ab-bbf0-fe6c859d6c9c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:14:35.598469 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.598443    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c3ad599-c376-47ab-bbf0-fe6c859d6c9c-maas-api-tls" pod "2c3ad599-c376-47ab-bbf0-fe6c859d6c9c" (UID: "2c3ad599-c376-47ab-bbf0-fe6c859d6c9c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:14:35.598583 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.598552    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c3ad599-c376-47ab-bbf0-fe6c859d6c9c-kube-api-access-9gk2j" pod "2c3ad599-c376-47ab-bbf0-fe6c859d6c9c" (UID: "2c3ad599-c376-47ab-bbf0-fe6c859d6c9c"). InnerVolumeSpecName "kube-api-access-9gk2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:35.633796 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.633676    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-68746cfdb7-hsxp2"
Oct 30 01:14:35.697534 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.697443    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a-plugins-config-volume\") pod \"8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a\" (UID: \"8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a\") "
Oct 30 01:14:35.697696 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.697555    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a-kube-api-access-4h4sf\" (UniqueName: \"kubernetes.io/projected/8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a-kube-api-access-4h4sf\") pod \"8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a\" (UID: \"8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a\") "
Oct 30 01:14:35.698826 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.698351    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gk2j\" (UniqueName: \"kubernetes.io/projected/2c3ad599-c376-47ab-bbf0-fe6c859d6c9c-kube-api-access-9gk2j\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:14:35.698826 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.698377    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c3ad599-c376-47ab-bbf0-fe6c859d6c9c-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:14:35.698826 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.698391    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c3ad599-c376-47ab-bbf0-fe6c859d6c9c-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:14:35.699516 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.699478    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a-plugins-config-volume" pod "8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a" (UID: "8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:14:35.702254 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.702228    4393 generic.go:356] "Generic (PLEG): container finished" podID="734513f7-51fc-49f0-98a4-bf801eb09fb1" containerID="7097715569e230b160ade9a20ca375c64c8ec117f1cd65d9832406ca8c125d46" exitCode=0
Oct 30 01:14:35.702447 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.702417    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4" event={"ID":"734513f7-51fc-49f0-98a4-bf801eb09fb1","Type":"ContainerDied","Data":"7097715569e230b160ade9a20ca375c64c8ec117f1cd65d9832406ca8c125d46"}
Oct 30 01:14:35.703262 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.703227    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a-kube-api-access-4h4sf" pod "8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a" (UID: "8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a"). InnerVolumeSpecName "kube-api-access-4h4sf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:35.704279 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.704256    4393 generic.go:356] "Generic (PLEG): container finished" podID="8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a" containerID="8e2c4f087ae3925b97cf39a9a161436fda2755954ebc485e501479d19a4f660e" exitCode=0
Oct 30 01:14:35.704427 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.704413    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-68746cfdb7-hsxp2"
Oct 30 01:14:35.705481 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.704896    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-68746cfdb7-hsxp2" event={"ID":"8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a","Type":"ContainerDied","Data":"8e2c4f087ae3925b97cf39a9a161436fda2755954ebc485e501479d19a4f660e"}
Oct 30 01:14:35.705481 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.704956    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-68746cfdb7-hsxp2" event={"ID":"8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a","Type":"ContainerDied","Data":"e8aa273278c405a09ec87104c1100a2fadd94334fd33f37812d1c1f50f73eb1d"}
Oct 30 01:14:35.705481 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.704985    4393 scope.go:122] "RemoveContainer" containerID="8e2c4f087ae3925b97cf39a9a161436fda2755954ebc485e501479d19a4f660e"
Oct 30 01:14:35.713853 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.709983    4393 generic.go:356] "Generic (PLEG): container finished" podID="2c3ad599-c376-47ab-bbf0-fe6c859d6c9c" containerID="0f31035d565898cfe4448ce2da149ab4d1959d4c1ba6916bbfccff7a98f021c3" exitCode=0
Oct 30 01:14:35.713853 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.710116    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb" event={"ID":"2c3ad599-c376-47ab-bbf0-fe6c859d6c9c","Type":"ContainerDied","Data":"0f31035d565898cfe4448ce2da149ab4d1959d4c1ba6916bbfccff7a98f021c3"}
Oct 30 01:14:35.713853 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.710147    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb" event={"ID":"2c3ad599-c376-47ab-bbf0-fe6c859d6c9c","Type":"ContainerDied","Data":"97205c0bc0d2cdedf5102c0c3a20de3775c785e106ecd48c26c88b3144722446"}
Oct 30 01:14:35.713853 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.710205    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb"
Oct 30 01:14:35.727967 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.726649    4393 scope.go:122] "RemoveContainer" containerID="8e2c4f087ae3925b97cf39a9a161436fda2755954ebc485e501479d19a4f660e"
Oct 30 01:14:35.732649 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:14:35.732596    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e2c4f087ae3925b97cf39a9a161436fda2755954ebc485e501479d19a4f660e\": container with ID starting with 8e2c4f087ae3925b97cf39a9a161436fda2755954ebc485e501479d19a4f660e not found: ID does not exist" containerID="8e2c4f087ae3925b97cf39a9a161436fda2755954ebc485e501479d19a4f660e"
Oct 30 01:14:35.732734 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.732638    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e2c4f087ae3925b97cf39a9a161436fda2755954ebc485e501479d19a4f660e"} err="failed to get container status \"8e2c4f087ae3925b97cf39a9a161436fda2755954ebc485e501479d19a4f660e\": rpc error: code = NotFound desc = could not find container \"8e2c4f087ae3925b97cf39a9a161436fda2755954ebc485e501479d19a4f660e\": container with ID starting with 8e2c4f087ae3925b97cf39a9a161436fda2755954ebc485e501479d19a4f660e not found: ID does not exist"
Oct 30 01:14:35.732734 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.732661    4393 scope.go:122] "RemoveContainer" containerID="0f31035d565898cfe4448ce2da149ab4d1959d4c1ba6916bbfccff7a98f021c3"
Oct 30 01:14:35.760035 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.760008    4393 scope.go:122] "RemoveContainer" containerID="0f31035d565898cfe4448ce2da149ab4d1959d4c1ba6916bbfccff7a98f021c3"
Oct 30 01:14:35.765310 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:14:35.765275    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f31035d565898cfe4448ce2da149ab4d1959d4c1ba6916bbfccff7a98f021c3\": container with ID starting with 0f31035d565898cfe4448ce2da149ab4d1959d4c1ba6916bbfccff7a98f021c3 not found: ID does not exist" containerID="0f31035d565898cfe4448ce2da149ab4d1959d4c1ba6916bbfccff7a98f021c3"
Oct 30 01:14:35.765463 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.765319    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f31035d565898cfe4448ce2da149ab4d1959d4c1ba6916bbfccff7a98f021c3"} err="failed to get container status \"0f31035d565898cfe4448ce2da149ab4d1959d4c1ba6916bbfccff7a98f021c3\": rpc error: code = NotFound desc = could not find container \"0f31035d565898cfe4448ce2da149ab4d1959d4c1ba6916bbfccff7a98f021c3\": container with ID starting with 0f31035d565898cfe4448ce2da149ab4d1959d4c1ba6916bbfccff7a98f021c3 not found: ID does not exist"
Oct 30 01:14:35.800350 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.789412    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-68746cfdb7-hsxp2"]
Oct 30 01:14:35.800350 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.796770    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-73f667-68746cfdb7-hsxp2"]
Oct 30 01:14:35.801642 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.801605    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:14:35.801815 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.801645    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h4sf\" (UniqueName: \"kubernetes.io/projected/8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a-kube-api-access-4h4sf\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:14:35.824234 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.824202    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb"]
Oct 30 01:14:35.834596 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.834561    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-73f667-6f7488f44-tmhsb"]
Oct 30 01:14:35.839212 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.839189    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4"
Oct 30 01:14:35.902630 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.902596    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/734513f7-51fc-49f0-98a4-bf801eb09fb1-kube-api-access-w2wlx\" (UniqueName: \"kubernetes.io/projected/734513f7-51fc-49f0-98a4-bf801eb09fb1-kube-api-access-w2wlx\") pod \"734513f7-51fc-49f0-98a4-bf801eb09fb1\" (UID: \"734513f7-51fc-49f0-98a4-bf801eb09fb1\") "
Oct 30 01:14:35.902869 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.902700    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/734513f7-51fc-49f0-98a4-bf801eb09fb1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/734513f7-51fc-49f0-98a4-bf801eb09fb1-openshift-service-ca\") pod \"734513f7-51fc-49f0-98a4-bf801eb09fb1\" (UID: \"734513f7-51fc-49f0-98a4-bf801eb09fb1\") "
Oct 30 01:14:35.902967 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.902871    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/734513f7-51fc-49f0-98a4-bf801eb09fb1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/734513f7-51fc-49f0-98a4-bf801eb09fb1-metrics-certs\") pod \"734513f7-51fc-49f0-98a4-bf801eb09fb1\" (UID: \"734513f7-51fc-49f0-98a4-bf801eb09fb1\") "
Oct 30 01:14:35.902967 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.902961    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/734513f7-51fc-49f0-98a4-bf801eb09fb1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/734513f7-51fc-49f0-98a4-bf801eb09fb1-maas-api-tls\") pod \"734513f7-51fc-49f0-98a4-bf801eb09fb1\" (UID: \"734513f7-51fc-49f0-98a4-bf801eb09fb1\") "
Oct 30 01:14:35.903086 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.903064    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/734513f7-51fc-49f0-98a4-bf801eb09fb1-openshift-service-ca" pod "734513f7-51fc-49f0-98a4-bf801eb09fb1" (UID: "734513f7-51fc-49f0-98a4-bf801eb09fb1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:14:35.903534 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.903508    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/734513f7-51fc-49f0-98a4-bf801eb09fb1-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:14:35.905103 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.905042    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/734513f7-51fc-49f0-98a4-bf801eb09fb1-kube-api-access-w2wlx" pod "734513f7-51fc-49f0-98a4-bf801eb09fb1" (UID: "734513f7-51fc-49f0-98a4-bf801eb09fb1"). InnerVolumeSpecName "kube-api-access-w2wlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:14:35.905185 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.905100    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/734513f7-51fc-49f0-98a4-bf801eb09fb1-maas-api-tls" pod "734513f7-51fc-49f0-98a4-bf801eb09fb1" (UID: "734513f7-51fc-49f0-98a4-bf801eb09fb1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:14:35.905185 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:35.905119    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/734513f7-51fc-49f0-98a4-bf801eb09fb1-metrics-certs" pod "734513f7-51fc-49f0-98a4-bf801eb09fb1" (UID: "734513f7-51fc-49f0-98a4-bf801eb09fb1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:14:36.008708 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:36.008617    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/734513f7-51fc-49f0-98a4-bf801eb09fb1-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:14:36.008999 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:36.008981    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2wlx\" (UniqueName: \"kubernetes.io/projected/734513f7-51fc-49f0-98a4-bf801eb09fb1-kube-api-access-w2wlx\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:14:36.012983 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:36.012953    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/734513f7-51fc-49f0-98a4-bf801eb09fb1-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:14:36.721557 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:36.721524    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4"
Oct 30 01:14:36.722088 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:36.721561    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4" event={"ID":"734513f7-51fc-49f0-98a4-bf801eb09fb1","Type":"ContainerDied","Data":"f8ded8b16aae98f85fa386bd43f6ddfb52773e30341941c266d4e0b83cac35da"}
Oct 30 01:14:36.722088 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:36.721627    4393 scope.go:122] "RemoveContainer" containerID="7097715569e230b160ade9a20ca375c64c8ec117f1cd65d9832406ca8c125d46"
Oct 30 01:14:36.773772 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:36.773746    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4"]
Oct 30 01:14:36.785258 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:36.785230    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-55a4c7-5b9bd46d88-66jm4"]
Oct 30 01:14:37.728214 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:37.728165    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c3ad599-c376-47ab-bbf0-fe6c859d6c9c" path="/var/lib/kubelet/pods/2c3ad599-c376-47ab-bbf0-fe6c859d6c9c/volumes"
Oct 30 01:14:37.728856 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:37.728835    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="734513f7-51fc-49f0-98a4-bf801eb09fb1" path="/var/lib/kubelet/pods/734513f7-51fc-49f0-98a4-bf801eb09fb1/volumes"
Oct 30 01:14:37.729401 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:14:37.729377    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a" path="/var/lib/kubelet/pods/8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a/volumes"
Oct 30 01:14:56.512989 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:14:56.512945    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:15:05.907255 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:05.907213    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt"]
Oct 30 01:15:05.908221 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:05.908161    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt" podUID="ecb5a5fb-3709-40c0-9ae4-b699d51ec79d" containerName="maas-api" containerID="cri-o://0ca5792c2281a7c4824c839d7d5b349495965e2896c8400cdfd1cac5ca1374ec" gracePeriod=30
Oct 30 01:15:05.965171 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:05.965137    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt"]
Oct 30 01:15:06.222734 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:06.222701    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt"
Oct 30 01:15:06.335055 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:06.335011    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecb5a5fb-3709-40c0-9ae4-b699d51ec79d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecb5a5fb-3709-40c0-9ae4-b699d51ec79d-maas-api-tls\") pod \"ecb5a5fb-3709-40c0-9ae4-b699d51ec79d\" (UID: \"ecb5a5fb-3709-40c0-9ae4-b699d51ec79d\") "
Oct 30 01:15:06.335255 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:06.335127    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecb5a5fb-3709-40c0-9ae4-b699d51ec79d-kube-api-access-jpkrb\" (UniqueName: \"kubernetes.io/projected/ecb5a5fb-3709-40c0-9ae4-b699d51ec79d-kube-api-access-jpkrb\") pod \"ecb5a5fb-3709-40c0-9ae4-b699d51ec79d\" (UID: \"ecb5a5fb-3709-40c0-9ae4-b699d51ec79d\") "
Oct 30 01:15:06.335255 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:06.335233    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecb5a5fb-3709-40c0-9ae4-b699d51ec79d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecb5a5fb-3709-40c0-9ae4-b699d51ec79d-openshift-service-ca\") pod \"ecb5a5fb-3709-40c0-9ae4-b699d51ec79d\" (UID: \"ecb5a5fb-3709-40c0-9ae4-b699d51ec79d\") "
Oct 30 01:15:06.335405 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:06.335302    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecb5a5fb-3709-40c0-9ae4-b699d51ec79d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecb5a5fb-3709-40c0-9ae4-b699d51ec79d-metrics-certs\") pod \"ecb5a5fb-3709-40c0-9ae4-b699d51ec79d\" (UID: \"ecb5a5fb-3709-40c0-9ae4-b699d51ec79d\") "
Oct 30 01:15:06.335698 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:06.335669    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecb5a5fb-3709-40c0-9ae4-b699d51ec79d-openshift-service-ca" pod "ecb5a5fb-3709-40c0-9ae4-b699d51ec79d" (UID: "ecb5a5fb-3709-40c0-9ae4-b699d51ec79d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:06.336117 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:06.336095    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecb5a5fb-3709-40c0-9ae4-b699d51ec79d-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:15:06.337858 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:06.337826    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecb5a5fb-3709-40c0-9ae4-b699d51ec79d-metrics-certs" pod "ecb5a5fb-3709-40c0-9ae4-b699d51ec79d" (UID: "ecb5a5fb-3709-40c0-9ae4-b699d51ec79d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:15:06.338326 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:06.338302    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecb5a5fb-3709-40c0-9ae4-b699d51ec79d-maas-api-tls" pod "ecb5a5fb-3709-40c0-9ae4-b699d51ec79d" (UID: "ecb5a5fb-3709-40c0-9ae4-b699d51ec79d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:15:06.339338 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:06.339316    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb5a5fb-3709-40c0-9ae4-b699d51ec79d-kube-api-access-jpkrb" pod "ecb5a5fb-3709-40c0-9ae4-b699d51ec79d" (UID: "ecb5a5fb-3709-40c0-9ae4-b699d51ec79d"). InnerVolumeSpecName "kube-api-access-jpkrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:06.449858 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:06.447144    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecb5a5fb-3709-40c0-9ae4-b699d51ec79d-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:15:06.449858 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:06.447184    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpkrb\" (UniqueName: \"kubernetes.io/projected/ecb5a5fb-3709-40c0-9ae4-b699d51ec79d-kube-api-access-jpkrb\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:15:06.449858 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:06.447201    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecb5a5fb-3709-40c0-9ae4-b699d51ec79d-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:15:07.017749 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:07.017708    4393 generic.go:356] "Generic (PLEG): container finished" podID="ecb5a5fb-3709-40c0-9ae4-b699d51ec79d" containerID="0ca5792c2281a7c4824c839d7d5b349495965e2896c8400cdfd1cac5ca1374ec" exitCode=0
Oct 30 01:15:07.018302 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:07.017772    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt"
Oct 30 01:15:07.018302 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:07.017808    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt" event={"ID":"ecb5a5fb-3709-40c0-9ae4-b699d51ec79d","Type":"ContainerDied","Data":"0ca5792c2281a7c4824c839d7d5b349495965e2896c8400cdfd1cac5ca1374ec"}
Oct 30 01:15:07.018302 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:07.017864    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt" event={"ID":"ecb5a5fb-3709-40c0-9ae4-b699d51ec79d","Type":"ContainerDied","Data":"df21017f86179e2bed2a35a568d03e86a57bbd9fe65cbd36ec19cdf3e3f3b18f"}
Oct 30 01:15:07.018302 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:07.017891    4393 scope.go:122] "RemoveContainer" containerID="0ca5792c2281a7c4824c839d7d5b349495965e2896c8400cdfd1cac5ca1374ec"
Oct 30 01:15:07.031406 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:07.031385    4393 scope.go:122] "RemoveContainer" containerID="0ca5792c2281a7c4824c839d7d5b349495965e2896c8400cdfd1cac5ca1374ec"
Oct 30 01:15:07.031689 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:15:07.031659    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ca5792c2281a7c4824c839d7d5b349495965e2896c8400cdfd1cac5ca1374ec\": container with ID starting with 0ca5792c2281a7c4824c839d7d5b349495965e2896c8400cdfd1cac5ca1374ec not found: ID does not exist" containerID="0ca5792c2281a7c4824c839d7d5b349495965e2896c8400cdfd1cac5ca1374ec"
Oct 30 01:15:07.031823 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:07.031695    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ca5792c2281a7c4824c839d7d5b349495965e2896c8400cdfd1cac5ca1374ec"} err="failed to get container status \"0ca5792c2281a7c4824c839d7d5b349495965e2896c8400cdfd1cac5ca1374ec\": rpc error: code = NotFound desc = could not find container \"0ca5792c2281a7c4824c839d7d5b349495965e2896c8400cdfd1cac5ca1374ec\": container with ID starting with 0ca5792c2281a7c4824c839d7d5b349495965e2896c8400cdfd1cac5ca1374ec not found: ID does not exist"
Oct 30 01:15:07.072043 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:07.072011    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt"]
Oct 30 01:15:07.080947 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:07.080916    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8b2c6-7f76c7c8c-bf8wt"]
Oct 30 01:15:07.726661 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:07.726629    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecb5a5fb-3709-40c0-9ae4-b699d51ec79d" path="/var/lib/kubelet/pods/ecb5a5fb-3709-40c0-9ae4-b699d51ec79d/volumes"
Oct 30 01:15:20.997366 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:20.997307    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt" podUID="1b3371ba-4d68-47a2-b45d-ef2470346e48" containerName="main" containerID="cri-o://178ead6272f08b75482d71381d3b6d6dfa2811d75d8c530769b6208542f13749" gracePeriod=45
Oct 30 01:15:21.307843 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:21.307761    4393 generic.go:356] "Generic (PLEG): container finished" podID="1b3371ba-4d68-47a2-b45d-ef2470346e48" containerID="178ead6272f08b75482d71381d3b6d6dfa2811d75d8c530769b6208542f13749" exitCode=0
Oct 30 01:15:21.308188 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:21.307846    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt" event={"ID":"1b3371ba-4d68-47a2-b45d-ef2470346e48","Type":"ContainerDied","Data":"178ead6272f08b75482d71381d3b6d6dfa2811d75d8c530769b6208542f13749"}
Oct 30 01:15:21.308379 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:21.308363    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt" event={"ID":"1b3371ba-4d68-47a2-b45d-ef2470346e48","Type":"ContainerDied","Data":"d7b961290840b97e868e5ad6f91f9cf4b3d59d103fb7312d3d6ae8542ef00b46"}
Oct 30 01:15:21.308480 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:21.308467    4393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7b961290840b97e868e5ad6f91f9cf4b3d59d103fb7312d3d6ae8542ef00b46"
Oct 30 01:15:21.319167 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:21.319136    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt"
Oct 30 01:15:21.446370 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:21.446340    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b3371ba-4d68-47a2-b45d-ef2470346e48-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b3371ba-4d68-47a2-b45d-ef2470346e48-dshm\") pod \"1b3371ba-4d68-47a2-b45d-ef2470346e48\" (UID: \"1b3371ba-4d68-47a2-b45d-ef2470346e48\") "
Oct 30 01:15:21.446573 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:21.446452    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b3371ba-4d68-47a2-b45d-ef2470346e48-tls-certs\" (UniqueName: \"kubernetes.io/secret/1b3371ba-4d68-47a2-b45d-ef2470346e48-tls-certs\") pod \"1b3371ba-4d68-47a2-b45d-ef2470346e48\" (UID: \"1b3371ba-4d68-47a2-b45d-ef2470346e48\") "
Oct 30 01:15:21.446573 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:21.446483    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b3371ba-4d68-47a2-b45d-ef2470346e48-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b3371ba-4d68-47a2-b45d-ef2470346e48-tmp-dir\") pod \"1b3371ba-4d68-47a2-b45d-ef2470346e48\" (UID: \"1b3371ba-4d68-47a2-b45d-ef2470346e48\") "
Oct 30 01:15:21.446573 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:21.446513    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b3371ba-4d68-47a2-b45d-ef2470346e48-kube-api-access-7gmj9\" (UniqueName: \"kubernetes.io/projected/1b3371ba-4d68-47a2-b45d-ef2470346e48-kube-api-access-7gmj9\") pod \"1b3371ba-4d68-47a2-b45d-ef2470346e48\" (UID: \"1b3371ba-4d68-47a2-b45d-ef2470346e48\") "
Oct 30 01:15:21.446796 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:21.446614    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b3371ba-4d68-47a2-b45d-ef2470346e48-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b3371ba-4d68-47a2-b45d-ef2470346e48-model-cache\") pod \"1b3371ba-4d68-47a2-b45d-ef2470346e48\" (UID: \"1b3371ba-4d68-47a2-b45d-ef2470346e48\") "
Oct 30 01:15:21.446959 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:21.446939    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b3371ba-4d68-47a2-b45d-ef2470346e48-home\" (UniqueName: \"kubernetes.io/empty-dir/1b3371ba-4d68-47a2-b45d-ef2470346e48-home\") pod \"1b3371ba-4d68-47a2-b45d-ef2470346e48\" (UID: \"1b3371ba-4d68-47a2-b45d-ef2470346e48\") "
Oct 30 01:15:21.447235 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:21.447126    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b3371ba-4d68-47a2-b45d-ef2470346e48-model-cache" pod "1b3371ba-4d68-47a2-b45d-ef2470346e48" (UID: "1b3371ba-4d68-47a2-b45d-ef2470346e48"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:21.447329 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:21.447225    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b3371ba-4d68-47a2-b45d-ef2470346e48-tmp-dir" pod "1b3371ba-4d68-47a2-b45d-ef2470346e48" (UID: "1b3371ba-4d68-47a2-b45d-ef2470346e48"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:21.447446 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:21.447418    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b3371ba-4d68-47a2-b45d-ef2470346e48-home" pod "1b3371ba-4d68-47a2-b45d-ef2470346e48" (UID: "1b3371ba-4d68-47a2-b45d-ef2470346e48"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:21.447761 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:21.447739    4393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b3371ba-4d68-47a2-b45d-ef2470346e48-model-cache\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:15:21.447898 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:21.447767    4393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b3371ba-4d68-47a2-b45d-ef2470346e48-home\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:15:21.447898 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:21.447822    4393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b3371ba-4d68-47a2-b45d-ef2470346e48-tmp-dir\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:15:21.456926 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:21.456895    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b3371ba-4d68-47a2-b45d-ef2470346e48-tls-certs" pod "1b3371ba-4d68-47a2-b45d-ef2470346e48" (UID: "1b3371ba-4d68-47a2-b45d-ef2470346e48"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:15:21.457065 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:21.456963    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b3371ba-4d68-47a2-b45d-ef2470346e48-kube-api-access-7gmj9" pod "1b3371ba-4d68-47a2-b45d-ef2470346e48" (UID: "1b3371ba-4d68-47a2-b45d-ef2470346e48"). InnerVolumeSpecName "kube-api-access-7gmj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:21.457898 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:21.457877    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b3371ba-4d68-47a2-b45d-ef2470346e48-dshm" pod "1b3371ba-4d68-47a2-b45d-ef2470346e48" (UID: "1b3371ba-4d68-47a2-b45d-ef2470346e48"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:15:21.550044 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:21.549130    4393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b3371ba-4d68-47a2-b45d-ef2470346e48-dshm\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:15:21.550044 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:21.549161    4393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b3371ba-4d68-47a2-b45d-ef2470346e48-tls-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:15:21.550044 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:21.549178    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gmj9\" (UniqueName: \"kubernetes.io/projected/1b3371ba-4d68-47a2-b45d-ef2470346e48-kube-api-access-7gmj9\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:15:22.330559 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:22.330519    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt"
Oct 30 01:15:22.393536 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:22.393502    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt"]
Oct 30 01:15:22.404345 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:22.404314    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-73435a/sub-test-model-73435a-kserve-57db64f59-vrxlt"]
Oct 30 01:15:23.727064 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:23.727035    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b3371ba-4d68-47a2-b45d-ef2470346e48" path="/var/lib/kubelet/pods/1b3371ba-4d68-47a2-b45d-ef2470346e48/volumes"
Oct 30 01:15:25.679865 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:25.679828    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5f387c-d78d85d96-f5csl"]
Oct 30 01:15:25.711539 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:25.711505    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5f387c-5cf86f7b99-9gvnl"]
Oct 30 01:15:26.519344 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:15:26.519304    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:15:30.469825 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:30.469765    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w"]
Oct 30 01:15:30.470333 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:30.470021    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w" podUID="cf0d8c60-1725-48a4-b3e6-bfc144e13031" containerName="maas-api" containerID="cri-o://b6d068ed946beb76dc52c9fcb834bd2ff685b46a04691da17ce5f58e220cd60a" gracePeriod=30
Oct 30 01:15:30.521392 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:30.521346    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w" podUID="cf0d8c60-1725-48a4-b3e6-bfc144e13031" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.113:8443/health\": dial tcp 10.133.0.113:8443: connect: connection refused"
Oct 30 01:15:30.680490 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:30.680416    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5f387c-d78d85d96-f5csl" podUID="1bc74d4c-96b2-4227-9c2b-1145739bab41" containerName="payload-processing" containerID="cri-o://5e2c3f0acce6ee3db982c54d892c648e624080b6f0bd47ca3685fc729aaf9a3f" gracePeriod=25
Oct 30 01:15:30.727805 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:30.719941    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5f387c-5cf86f7b99-9gvnl" podUID="048e31e4-12d0-46ef-abae-1afebf63da75" containerName="payload-pre-processing" containerID="cri-o://a10799c3c84c835c64b9a96de5aad694f99cf956443643a1a5c598da9c69e99d" gracePeriod=25
Oct 30 01:15:31.013547 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.012995    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w"
Oct 30 01:15:31.135499 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.135467    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf0d8c60-1725-48a4-b3e6-bfc144e13031-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf0d8c60-1725-48a4-b3e6-bfc144e13031-metrics-certs\") pod \"cf0d8c60-1725-48a4-b3e6-bfc144e13031\" (UID: \"cf0d8c60-1725-48a4-b3e6-bfc144e13031\") "
Oct 30 01:15:31.147230 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.147161    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf0d8c60-1725-48a4-b3e6-bfc144e13031-metrics-certs" pod "cf0d8c60-1725-48a4-b3e6-bfc144e13031" (UID: "cf0d8c60-1725-48a4-b3e6-bfc144e13031"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:15:31.151516 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.149929    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf0d8c60-1725-48a4-b3e6-bfc144e13031-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf0d8c60-1725-48a4-b3e6-bfc144e13031-openshift-service-ca\") pod \"cf0d8c60-1725-48a4-b3e6-bfc144e13031\" (UID: \"cf0d8c60-1725-48a4-b3e6-bfc144e13031\") "
Oct 30 01:15:31.151516 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.150007    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf0d8c60-1725-48a4-b3e6-bfc144e13031-kube-api-access-xjqld\" (UniqueName: \"kubernetes.io/projected/cf0d8c60-1725-48a4-b3e6-bfc144e13031-kube-api-access-xjqld\") pod \"cf0d8c60-1725-48a4-b3e6-bfc144e13031\" (UID: \"cf0d8c60-1725-48a4-b3e6-bfc144e13031\") "
Oct 30 01:15:31.151516 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.150144    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf0d8c60-1725-48a4-b3e6-bfc144e13031-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf0d8c60-1725-48a4-b3e6-bfc144e13031-maas-api-tls\") pod \"cf0d8c60-1725-48a4-b3e6-bfc144e13031\" (UID: \"cf0d8c60-1725-48a4-b3e6-bfc144e13031\") "
Oct 30 01:15:31.151516 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.151044    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf0d8c60-1725-48a4-b3e6-bfc144e13031-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:15:31.151516 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.151485    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf0d8c60-1725-48a4-b3e6-bfc144e13031-openshift-service-ca" pod "cf0d8c60-1725-48a4-b3e6-bfc144e13031" (UID: "cf0d8c60-1725-48a4-b3e6-bfc144e13031"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:31.160902 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.160873    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf0d8c60-1725-48a4-b3e6-bfc144e13031-maas-api-tls" pod "cf0d8c60-1725-48a4-b3e6-bfc144e13031" (UID: "cf0d8c60-1725-48a4-b3e6-bfc144e13031"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:15:31.161510 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.161475    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf0d8c60-1725-48a4-b3e6-bfc144e13031-kube-api-access-xjqld" pod "cf0d8c60-1725-48a4-b3e6-bfc144e13031" (UID: "cf0d8c60-1725-48a4-b3e6-bfc144e13031"). InnerVolumeSpecName "kube-api-access-xjqld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:31.193870 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.193844    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5f387c-d78d85d96-f5csl"
Oct 30 01:15:31.255847 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.255086    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bc74d4c-96b2-4227-9c2b-1145739bab41-kube-api-access-c2bb7\" (UniqueName: \"kubernetes.io/projected/1bc74d4c-96b2-4227-9c2b-1145739bab41-kube-api-access-c2bb7\") pod \"1bc74d4c-96b2-4227-9c2b-1145739bab41\" (UID: \"1bc74d4c-96b2-4227-9c2b-1145739bab41\") "
Oct 30 01:15:31.255847 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.255195    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bc74d4c-96b2-4227-9c2b-1145739bab41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bc74d4c-96b2-4227-9c2b-1145739bab41-plugins-config-volume\") pod \"1bc74d4c-96b2-4227-9c2b-1145739bab41\" (UID: \"1bc74d4c-96b2-4227-9c2b-1145739bab41\") "
Oct 30 01:15:31.255847 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.255773    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf0d8c60-1725-48a4-b3e6-bfc144e13031-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:15:31.255847 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.255808    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf0d8c60-1725-48a4-b3e6-bfc144e13031-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:15:31.255847 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.255819    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjqld\" (UniqueName: \"kubernetes.io/projected/cf0d8c60-1725-48a4-b3e6-bfc144e13031-kube-api-access-xjqld\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:15:31.256282 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.256253    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bc74d4c-96b2-4227-9c2b-1145739bab41-plugins-config-volume" pod "1bc74d4c-96b2-4227-9c2b-1145739bab41" (UID: "1bc74d4c-96b2-4227-9c2b-1145739bab41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:31.258386 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.258363    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bc74d4c-96b2-4227-9c2b-1145739bab41-kube-api-access-c2bb7" pod "1bc74d4c-96b2-4227-9c2b-1145739bab41" (UID: "1bc74d4c-96b2-4227-9c2b-1145739bab41"). InnerVolumeSpecName "kube-api-access-c2bb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:31.296645 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.296619    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5f387c-5cf86f7b99-9gvnl"
Oct 30 01:15:31.357300 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.357259    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/048e31e4-12d0-46ef-abae-1afebf63da75-kube-api-access-4hm9l\" (UniqueName: \"kubernetes.io/projected/048e31e4-12d0-46ef-abae-1afebf63da75-kube-api-access-4hm9l\") pod \"048e31e4-12d0-46ef-abae-1afebf63da75\" (UID: \"048e31e4-12d0-46ef-abae-1afebf63da75\") "
Oct 30 01:15:31.357495 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.357339    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/048e31e4-12d0-46ef-abae-1afebf63da75-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048e31e4-12d0-46ef-abae-1afebf63da75-plugins-config-volume\") pod \"048e31e4-12d0-46ef-abae-1afebf63da75\" (UID: \"048e31e4-12d0-46ef-abae-1afebf63da75\") "
Oct 30 01:15:31.357853 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.357822    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/048e31e4-12d0-46ef-abae-1afebf63da75-plugins-config-volume" pod "048e31e4-12d0-46ef-abae-1afebf63da75" (UID: "048e31e4-12d0-46ef-abae-1afebf63da75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:31.358286 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.358263    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bc74d4c-96b2-4227-9c2b-1145739bab41-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:15:31.358394 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.358293    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048e31e4-12d0-46ef-abae-1afebf63da75-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:15:31.358394 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.358308    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2bb7\" (UniqueName: \"kubernetes.io/projected/1bc74d4c-96b2-4227-9c2b-1145739bab41-kube-api-access-c2bb7\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:15:31.360022 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.359990    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/048e31e4-12d0-46ef-abae-1afebf63da75-kube-api-access-4hm9l" pod "048e31e4-12d0-46ef-abae-1afebf63da75" (UID: "048e31e4-12d0-46ef-abae-1afebf63da75"). InnerVolumeSpecName "kube-api-access-4hm9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:31.415835 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.415806    4393 generic.go:356] "Generic (PLEG): container finished" podID="048e31e4-12d0-46ef-abae-1afebf63da75" containerID="a10799c3c84c835c64b9a96de5aad694f99cf956443643a1a5c598da9c69e99d" exitCode=0
Oct 30 01:15:31.415987 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.415885    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5f387c-5cf86f7b99-9gvnl" event={"ID":"048e31e4-12d0-46ef-abae-1afebf63da75","Type":"ContainerDied","Data":"a10799c3c84c835c64b9a96de5aad694f99cf956443643a1a5c598da9c69e99d"}
Oct 30 01:15:31.415987 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.415914    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5f387c-5cf86f7b99-9gvnl"
Oct 30 01:15:31.415987 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.415947    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5f387c-5cf86f7b99-9gvnl" event={"ID":"048e31e4-12d0-46ef-abae-1afebf63da75","Type":"ContainerDied","Data":"7ab1f73a768871b9c309be479fdd8ffd86738df1c688e0f7fc2db631af55abfb"}
Oct 30 01:15:31.415987 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.415970    4393 scope.go:122] "RemoveContainer" containerID="a10799c3c84c835c64b9a96de5aad694f99cf956443643a1a5c598da9c69e99d"
Oct 30 01:15:31.417379 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.417356    4393 generic.go:356] "Generic (PLEG): container finished" podID="cf0d8c60-1725-48a4-b3e6-bfc144e13031" containerID="b6d068ed946beb76dc52c9fcb834bd2ff685b46a04691da17ce5f58e220cd60a" exitCode=0
Oct 30 01:15:31.417483 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.417389    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w"
Oct 30 01:15:31.417483 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.417433    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w" event={"ID":"cf0d8c60-1725-48a4-b3e6-bfc144e13031","Type":"ContainerDied","Data":"b6d068ed946beb76dc52c9fcb834bd2ff685b46a04691da17ce5f58e220cd60a"}
Oct 30 01:15:31.417483 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.417480    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w" event={"ID":"cf0d8c60-1725-48a4-b3e6-bfc144e13031","Type":"ContainerDied","Data":"06659f9d793792e6bc710ccb9d059f5dbda92121ceefa56f3df3f27f02b4ff63"}
Oct 30 01:15:31.419168 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.419140    4393 generic.go:356] "Generic (PLEG): container finished" podID="1bc74d4c-96b2-4227-9c2b-1145739bab41" containerID="5e2c3f0acce6ee3db982c54d892c648e624080b6f0bd47ca3685fc729aaf9a3f" exitCode=0
Oct 30 01:15:31.419310 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.419184    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5f387c-d78d85d96-f5csl"
Oct 30 01:15:31.419310 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.419217    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5f387c-d78d85d96-f5csl" event={"ID":"1bc74d4c-96b2-4227-9c2b-1145739bab41","Type":"ContainerDied","Data":"5e2c3f0acce6ee3db982c54d892c648e624080b6f0bd47ca3685fc729aaf9a3f"}
Oct 30 01:15:31.419310 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.419273    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5f387c-d78d85d96-f5csl" event={"ID":"1bc74d4c-96b2-4227-9c2b-1145739bab41","Type":"ContainerDied","Data":"6a30224a470891a669f5ada881e6dee783a9404616eb5cd71779fb7246c54b0f"}
Oct 30 01:15:31.429012 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.428990    4393 scope.go:122] "RemoveContainer" containerID="a10799c3c84c835c64b9a96de5aad694f99cf956443643a1a5c598da9c69e99d"
Oct 30 01:15:31.429423 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:15:31.429279    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a10799c3c84c835c64b9a96de5aad694f99cf956443643a1a5c598da9c69e99d\": container with ID starting with a10799c3c84c835c64b9a96de5aad694f99cf956443643a1a5c598da9c69e99d not found: ID does not exist" containerID="a10799c3c84c835c64b9a96de5aad694f99cf956443643a1a5c598da9c69e99d"
Oct 30 01:15:31.429423 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.429315    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a10799c3c84c835c64b9a96de5aad694f99cf956443643a1a5c598da9c69e99d"} err="failed to get container status \"a10799c3c84c835c64b9a96de5aad694f99cf956443643a1a5c598da9c69e99d\": rpc error: code = NotFound desc = could not find container \"a10799c3c84c835c64b9a96de5aad694f99cf956443643a1a5c598da9c69e99d\": container with ID starting with a10799c3c84c835c64b9a96de5aad694f99cf956443643a1a5c598da9c69e99d not found: ID does not exist"
Oct 30 01:15:31.429423 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.429337    4393 scope.go:122] "RemoveContainer" containerID="b6d068ed946beb76dc52c9fcb834bd2ff685b46a04691da17ce5f58e220cd60a"
Oct 30 01:15:31.440341 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.440324    4393 scope.go:122] "RemoveContainer" containerID="b6d068ed946beb76dc52c9fcb834bd2ff685b46a04691da17ce5f58e220cd60a"
Oct 30 01:15:31.440653 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:15:31.440632    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6d068ed946beb76dc52c9fcb834bd2ff685b46a04691da17ce5f58e220cd60a\": container with ID starting with b6d068ed946beb76dc52c9fcb834bd2ff685b46a04691da17ce5f58e220cd60a not found: ID does not exist" containerID="b6d068ed946beb76dc52c9fcb834bd2ff685b46a04691da17ce5f58e220cd60a"
Oct 30 01:15:31.440706 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.440661    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6d068ed946beb76dc52c9fcb834bd2ff685b46a04691da17ce5f58e220cd60a"} err="failed to get container status \"b6d068ed946beb76dc52c9fcb834bd2ff685b46a04691da17ce5f58e220cd60a\": rpc error: code = NotFound desc = could not find container \"b6d068ed946beb76dc52c9fcb834bd2ff685b46a04691da17ce5f58e220cd60a\": container with ID starting with b6d068ed946beb76dc52c9fcb834bd2ff685b46a04691da17ce5f58e220cd60a not found: ID does not exist"
Oct 30 01:15:31.440706 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.440678    4393 scope.go:122] "RemoveContainer" containerID="5e2c3f0acce6ee3db982c54d892c648e624080b6f0bd47ca3685fc729aaf9a3f"
Oct 30 01:15:31.452005 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.451988    4393 scope.go:122] "RemoveContainer" containerID="5e2c3f0acce6ee3db982c54d892c648e624080b6f0bd47ca3685fc729aaf9a3f"
Oct 30 01:15:31.452305 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:15:31.452279    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e2c3f0acce6ee3db982c54d892c648e624080b6f0bd47ca3685fc729aaf9a3f\": container with ID starting with 5e2c3f0acce6ee3db982c54d892c648e624080b6f0bd47ca3685fc729aaf9a3f not found: ID does not exist" containerID="5e2c3f0acce6ee3db982c54d892c648e624080b6f0bd47ca3685fc729aaf9a3f"
Oct 30 01:15:31.452420 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.452309    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e2c3f0acce6ee3db982c54d892c648e624080b6f0bd47ca3685fc729aaf9a3f"} err="failed to get container status \"5e2c3f0acce6ee3db982c54d892c648e624080b6f0bd47ca3685fc729aaf9a3f\": rpc error: code = NotFound desc = could not find container \"5e2c3f0acce6ee3db982c54d892c648e624080b6f0bd47ca3685fc729aaf9a3f\": container with ID starting with 5e2c3f0acce6ee3db982c54d892c648e624080b6f0bd47ca3685fc729aaf9a3f not found: ID does not exist"
Oct 30 01:15:31.460332 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.460310    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hm9l\" (UniqueName: \"kubernetes.io/projected/048e31e4-12d0-46ef-abae-1afebf63da75-kube-api-access-4hm9l\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:15:31.484236 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.484213    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5f387c-5cf86f7b99-9gvnl"]
Oct 30 01:15:31.491398 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.491372    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5f387c-5cf86f7b99-9gvnl"]
Oct 30 01:15:31.538032 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.537843    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5f387c-d78d85d96-f5csl"]
Oct 30 01:15:31.563291 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.561565    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5f387c-d78d85d96-f5csl"]
Oct 30 01:15:31.591486 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.591250    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w"]
Oct 30 01:15:31.597772 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.597737    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5f387c-c657fccc5-t6s5w"]
Oct 30 01:15:31.726433 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.726397    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="048e31e4-12d0-46ef-abae-1afebf63da75" path="/var/lib/kubelet/pods/048e31e4-12d0-46ef-abae-1afebf63da75/volumes"
Oct 30 01:15:31.727085 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.727062    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bc74d4c-96b2-4227-9c2b-1145739bab41" path="/var/lib/kubelet/pods/1bc74d4c-96b2-4227-9c2b-1145739bab41/volumes"
Oct 30 01:15:31.727645 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:31.727629    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf0d8c60-1725-48a4-b3e6-bfc144e13031" path="/var/lib/kubelet/pods/cf0d8c60-1725-48a4-b3e6-bfc144e13031/volumes"
Oct 30 01:15:42.436417 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:42.436381    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj"]
Oct 30 01:15:42.436961 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:42.436621    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj" podUID="a0dd37a5-2f69-4f95-9e3e-958735302616" containerName="maas-api" containerID="cri-o://710c7da91e5cedc6165605df2b48fa56bd928b0b2a2bf4a07e7d7ee19b6b9e14" gracePeriod=30
Oct 30 01:15:42.577929 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:42.577815    4393 generic.go:356] "Generic (PLEG): container finished" podID="a0dd37a5-2f69-4f95-9e3e-958735302616" containerID="710c7da91e5cedc6165605df2b48fa56bd928b0b2a2bf4a07e7d7ee19b6b9e14" exitCode=0
Oct 30 01:15:42.577929 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:42.577819    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj" event={"ID":"a0dd37a5-2f69-4f95-9e3e-958735302616","Type":"ContainerDied","Data":"710c7da91e5cedc6165605df2b48fa56bd928b0b2a2bf4a07e7d7ee19b6b9e14"}
Oct 30 01:15:42.753861 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:42.753833    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj"
Oct 30 01:15:42.881223 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:42.881191    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0dd37a5-2f69-4f95-9e3e-958735302616-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0dd37a5-2f69-4f95-9e3e-958735302616-openshift-service-ca\") pod \"a0dd37a5-2f69-4f95-9e3e-958735302616\" (UID: \"a0dd37a5-2f69-4f95-9e3e-958735302616\") "
Oct 30 01:15:42.881404 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:42.881313    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0dd37a5-2f69-4f95-9e3e-958735302616-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0dd37a5-2f69-4f95-9e3e-958735302616-metrics-certs\") pod \"a0dd37a5-2f69-4f95-9e3e-958735302616\" (UID: \"a0dd37a5-2f69-4f95-9e3e-958735302616\") "
Oct 30 01:15:42.881462 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:42.881414    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0dd37a5-2f69-4f95-9e3e-958735302616-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0dd37a5-2f69-4f95-9e3e-958735302616-maas-api-tls\") pod \"a0dd37a5-2f69-4f95-9e3e-958735302616\" (UID: \"a0dd37a5-2f69-4f95-9e3e-958735302616\") "
Oct 30 01:15:42.881519 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:42.881500    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0dd37a5-2f69-4f95-9e3e-958735302616-kube-api-access-5v8jg\" (UniqueName: \"kubernetes.io/projected/a0dd37a5-2f69-4f95-9e3e-958735302616-kube-api-access-5v8jg\") pod \"a0dd37a5-2f69-4f95-9e3e-958735302616\" (UID: \"a0dd37a5-2f69-4f95-9e3e-958735302616\") "
Oct 30 01:15:42.881581 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:42.881533    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0dd37a5-2f69-4f95-9e3e-958735302616-openshift-service-ca" pod "a0dd37a5-2f69-4f95-9e3e-958735302616" (UID: "a0dd37a5-2f69-4f95-9e3e-958735302616"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:42.882662 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:42.882394    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0dd37a5-2f69-4f95-9e3e-958735302616-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:15:42.883677 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:42.883652    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0dd37a5-2f69-4f95-9e3e-958735302616-maas-api-tls" pod "a0dd37a5-2f69-4f95-9e3e-958735302616" (UID: "a0dd37a5-2f69-4f95-9e3e-958735302616"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:15:42.883814 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:42.883677    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0dd37a5-2f69-4f95-9e3e-958735302616-metrics-certs" pod "a0dd37a5-2f69-4f95-9e3e-958735302616" (UID: "a0dd37a5-2f69-4f95-9e3e-958735302616"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:15:42.883814 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:42.883762    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0dd37a5-2f69-4f95-9e3e-958735302616-kube-api-access-5v8jg" pod "a0dd37a5-2f69-4f95-9e3e-958735302616" (UID: "a0dd37a5-2f69-4f95-9e3e-958735302616"). InnerVolumeSpecName "kube-api-access-5v8jg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:42.990854 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:42.986161    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0dd37a5-2f69-4f95-9e3e-958735302616-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:15:42.990854 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:42.986205    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0dd37a5-2f69-4f95-9e3e-958735302616-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:15:42.990854 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:42.986217    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v8jg\" (UniqueName: \"kubernetes.io/projected/a0dd37a5-2f69-4f95-9e3e-958735302616-kube-api-access-5v8jg\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:15:43.587454 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:43.587421    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj" event={"ID":"a0dd37a5-2f69-4f95-9e3e-958735302616","Type":"ContainerDied","Data":"1933f14665e937fb37d391d1219a2aa18907f462e0008696ae32a39f99bf6d7e"}
Oct 30 01:15:43.587454 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:43.587442    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj"
Oct 30 01:15:43.588020 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:43.587480    4393 scope.go:122] "RemoveContainer" containerID="710c7da91e5cedc6165605df2b48fa56bd928b0b2a2bf4a07e7d7ee19b6b9e14"
Oct 30 01:15:43.650675 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:43.650638    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj"]
Oct 30 01:15:43.662026 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:43.661984    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d44c2-59f8cbcd89-l95rj"]
Oct 30 01:15:43.726935 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:43.726898    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0dd37a5-2f69-4f95-9e3e-958735302616" path="/var/lib/kubelet/pods/a0dd37a5-2f69-4f95-9e3e-958735302616/volumes"
Oct 30 01:15:50.786369 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:50.786331    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk"]
Oct 30 01:15:50.786917 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:50.786575    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk" podUID="f72f3812-276c-4a44-af20-917bc1e32a3f" containerName="maas-api" containerID="cri-o://ceefd0a93fea8d5683bdc6344f55688cd3fc800a1db9815895b3fd3b6c623391" gracePeriod=30
Oct 30 01:15:51.308491 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:51.308008    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk"
Oct 30 01:15:51.401736 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:51.401696    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f72f3812-276c-4a44-af20-917bc1e32a3f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f72f3812-276c-4a44-af20-917bc1e32a3f-maas-api-tls\") pod \"f72f3812-276c-4a44-af20-917bc1e32a3f\" (UID: \"f72f3812-276c-4a44-af20-917bc1e32a3f\") "
Oct 30 01:15:51.401944 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:51.401896    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f72f3812-276c-4a44-af20-917bc1e32a3f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f72f3812-276c-4a44-af20-917bc1e32a3f-openshift-service-ca\") pod \"f72f3812-276c-4a44-af20-917bc1e32a3f\" (UID: \"f72f3812-276c-4a44-af20-917bc1e32a3f\") "
Oct 30 01:15:51.402170 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:51.402140    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f72f3812-276c-4a44-af20-917bc1e32a3f-kube-api-access-kxw5p\" (UniqueName: \"kubernetes.io/projected/f72f3812-276c-4a44-af20-917bc1e32a3f-kube-api-access-kxw5p\") pod \"f72f3812-276c-4a44-af20-917bc1e32a3f\" (UID: \"f72f3812-276c-4a44-af20-917bc1e32a3f\") "
Oct 30 01:15:51.402320 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:51.402278    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f72f3812-276c-4a44-af20-917bc1e32a3f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f72f3812-276c-4a44-af20-917bc1e32a3f-metrics-certs\") pod \"f72f3812-276c-4a44-af20-917bc1e32a3f\" (UID: \"f72f3812-276c-4a44-af20-917bc1e32a3f\") "
Oct 30 01:15:51.403369 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:51.403339    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f72f3812-276c-4a44-af20-917bc1e32a3f-openshift-service-ca" pod "f72f3812-276c-4a44-af20-917bc1e32a3f" (UID: "f72f3812-276c-4a44-af20-917bc1e32a3f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:15:51.405905 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:51.405198    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f72f3812-276c-4a44-af20-917bc1e32a3f-maas-api-tls" pod "f72f3812-276c-4a44-af20-917bc1e32a3f" (UID: "f72f3812-276c-4a44-af20-917bc1e32a3f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:15:51.406082 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:51.406057    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f72f3812-276c-4a44-af20-917bc1e32a3f-kube-api-access-kxw5p" pod "f72f3812-276c-4a44-af20-917bc1e32a3f" (UID: "f72f3812-276c-4a44-af20-917bc1e32a3f"). InnerVolumeSpecName "kube-api-access-kxw5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:15:51.406920 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:51.406884    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f72f3812-276c-4a44-af20-917bc1e32a3f-metrics-certs" pod "f72f3812-276c-4a44-af20-917bc1e32a3f" (UID: "f72f3812-276c-4a44-af20-917bc1e32a3f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:15:51.488292 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:51.488255    4393 scope.go:122] "RemoveContainer" containerID="ceefd0a93fea8d5683bdc6344f55688cd3fc800a1db9815895b3fd3b6c623391"
Oct 30 01:15:51.502892 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:51.502850    4393 scope.go:122] "RemoveContainer" containerID="45fd3fd7fd48861b334246daf47e6084e429dd675eb5c3f8ecc5466a4e3259e8"
Oct 30 01:15:51.504233 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:51.504211    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f72f3812-276c-4a44-af20-917bc1e32a3f-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:15:51.504304 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:51.504238    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f72f3812-276c-4a44-af20-917bc1e32a3f-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:15:51.504304 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:51.504253    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f72f3812-276c-4a44-af20-917bc1e32a3f-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:15:51.504304 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:51.504268    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxw5p\" (UniqueName: \"kubernetes.io/projected/f72f3812-276c-4a44-af20-917bc1e32a3f-kube-api-access-kxw5p\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:15:51.747109 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:51.747020    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk" event={"ID":"f72f3812-276c-4a44-af20-917bc1e32a3f","Type":"ContainerDied","Data":"ceefd0a93fea8d5683bdc6344f55688cd3fc800a1db9815895b3fd3b6c623391"}
Oct 30 01:15:51.747109 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:51.747082    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk" event={"ID":"f72f3812-276c-4a44-af20-917bc1e32a3f","Type":"ContainerDied","Data":"9eb49ef2827e7568b937619ecd6ffea16b36ea6a5695c8942aa56f6b79bad70c"}
Oct 30 01:15:51.747302 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:51.747107    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk"
Oct 30 01:15:51.824339 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:51.824308    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk"]
Oct 30 01:15:51.828645 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:51.828621    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-00e07a-68dd55d949-tpmzk"]
Oct 30 01:15:53.726634 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:53.726599    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f72f3812-276c-4a44-af20-917bc1e32a3f" path="/var/lib/kubelet/pods/f72f3812-276c-4a44-af20-917bc1e32a3f/volumes"
Oct 30 01:15:56.512325 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:15:56.512280    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:15:56.714425 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.714391    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2"]
Oct 30 01:15:56.715250 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.715233    4393 cpu_manager.go:414] "containerMap: removing container" podUID="cf0d8c60-1725-48a4-b3e6-bfc144e13031" containerName="maas-api"
Oct 30 01:15:56.715302 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.715252    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf0d8c60-1725-48a4-b3e6-bfc144e13031" containerName="maas-api"
Oct 30 01:15:56.715302 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.715265    4393 cpu_manager.go:414] "containerMap: removing container" podUID="f72f3812-276c-4a44-af20-917bc1e32a3f" containerName="maas-api"
Oct 30 01:15:56.715302 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.715270    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f72f3812-276c-4a44-af20-917bc1e32a3f" containerName="maas-api"
Oct 30 01:15:56.715302 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.715283    4393 cpu_manager.go:414] "containerMap: removing container" podUID="ecb5a5fb-3709-40c0-9ae4-b699d51ec79d" containerName="maas-api"
Oct 30 01:15:56.715302 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.715289    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecb5a5fb-3709-40c0-9ae4-b699d51ec79d" containerName="maas-api"
Oct 30 01:15:56.715302 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.715295    4393 cpu_manager.go:414] "containerMap: removing container" podUID="8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a" containerName="payload-pre-processing"
Oct 30 01:15:56.715302 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.715300    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a" containerName="payload-pre-processing"
Oct 30 01:15:56.715521 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.715314    4393 cpu_manager.go:414] "containerMap: removing container" podUID="1bc74d4c-96b2-4227-9c2b-1145739bab41" containerName="payload-processing"
Oct 30 01:15:56.715521 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.715319    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bc74d4c-96b2-4227-9c2b-1145739bab41" containerName="payload-processing"
Oct 30 01:15:56.715521 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.715333    4393 cpu_manager.go:414] "containerMap: removing container" podUID="2c3ad599-c376-47ab-bbf0-fe6c859d6c9c" containerName="maas-api"
Oct 30 01:15:56.715521 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.715337    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c3ad599-c376-47ab-bbf0-fe6c859d6c9c" containerName="maas-api"
Oct 30 01:15:56.715521 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.715351    4393 cpu_manager.go:414] "containerMap: removing container" podUID="734513f7-51fc-49f0-98a4-bf801eb09fb1" containerName="maas-api"
Oct 30 01:15:56.715521 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.715356    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="734513f7-51fc-49f0-98a4-bf801eb09fb1" containerName="maas-api"
Oct 30 01:15:56.715521 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.715368    4393 cpu_manager.go:414] "containerMap: removing container" podUID="1b3371ba-4d68-47a2-b45d-ef2470346e48" containerName="main"
Oct 30 01:15:56.715521 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.715373    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b3371ba-4d68-47a2-b45d-ef2470346e48" containerName="main"
Oct 30 01:15:56.715521 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.715380    4393 cpu_manager.go:414] "containerMap: removing container" podUID="048e31e4-12d0-46ef-abae-1afebf63da75" containerName="payload-pre-processing"
Oct 30 01:15:56.715521 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.715384    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="048e31e4-12d0-46ef-abae-1afebf63da75" containerName="payload-pre-processing"
Oct 30 01:15:56.715521 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.715407    4393 cpu_manager.go:414] "containerMap: removing container" podUID="a0dd37a5-2f69-4f95-9e3e-958735302616" containerName="maas-api"
Oct 30 01:15:56.715521 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.715411    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0dd37a5-2f69-4f95-9e3e-958735302616" containerName="maas-api"
Oct 30 01:15:56.715521 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.715519    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0dd37a5-2f69-4f95-9e3e-958735302616" containerName="maas-api"
Oct 30 01:15:56.715932 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.715530    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf0d8c60-1725-48a4-b3e6-bfc144e13031" containerName="maas-api"
Oct 30 01:15:56.715932 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.715538    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="f72f3812-276c-4a44-af20-917bc1e32a3f" containerName="maas-api"
Oct 30 01:15:56.715932 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.715545    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecb5a5fb-3709-40c0-9ae4-b699d51ec79d" containerName="maas-api"
Oct 30 01:15:56.715932 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.715550    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d81b7b6-1c1f-43dc-82c9-1ccc444a7b5a" containerName="payload-pre-processing"
Oct 30 01:15:56.715932 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.715557    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bc74d4c-96b2-4227-9c2b-1145739bab41" containerName="payload-processing"
Oct 30 01:15:56.715932 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.715563    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c3ad599-c376-47ab-bbf0-fe6c859d6c9c" containerName="maas-api"
Oct 30 01:15:56.715932 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.715571    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="734513f7-51fc-49f0-98a4-bf801eb09fb1" containerName="maas-api"
Oct 30 01:15:56.715932 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.715578    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b3371ba-4d68-47a2-b45d-ef2470346e48" containerName="main"
Oct 30 01:15:56.715932 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.715582    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="048e31e4-12d0-46ef-abae-1afebf63da75" containerName="payload-pre-processing"
Oct 30 01:15:56.721328 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.721303    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2"
Oct 30 01:15:56.740271 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.740237    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2"]
Oct 30 01:15:56.764057 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.745004    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f6d5b8-serving-cert\""
Oct 30 01:15:56.891407 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.891373    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trphh\" (UniqueName: \"kubernetes.io/projected/a002f857-7a6e-4fb6-a05c-bce70bac9342-kube-api-access-trphh\") pod \"maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2\" (UID: \"a002f857-7a6e-4fb6-a05c-bce70bac9342\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2"
Oct 30 01:15:56.891567 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.891534    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a002f857-7a6e-4fb6-a05c-bce70bac9342-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2\" (UID: \"a002f857-7a6e-4fb6-a05c-bce70bac9342\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2"
Oct 30 01:15:56.891652 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.891635    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a002f857-7a6e-4fb6-a05c-bce70bac9342-metrics-certs\") pod \"maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2\" (UID: \"a002f857-7a6e-4fb6-a05c-bce70bac9342\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2"
Oct 30 01:15:56.891723 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:56.891712    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a002f857-7a6e-4fb6-a05c-bce70bac9342-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2\" (UID: \"a002f857-7a6e-4fb6-a05c-bce70bac9342\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2"
Oct 30 01:15:57.015132 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:57.015039    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trphh\" (UniqueName: \"kubernetes.io/projected/a002f857-7a6e-4fb6-a05c-bce70bac9342-kube-api-access-trphh\") pod \"maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2\" (UID: \"a002f857-7a6e-4fb6-a05c-bce70bac9342\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2"
Oct 30 01:15:57.015544 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:57.015510    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a002f857-7a6e-4fb6-a05c-bce70bac9342-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2\" (UID: \"a002f857-7a6e-4fb6-a05c-bce70bac9342\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2"
Oct 30 01:15:57.015826 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:57.015755    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a002f857-7a6e-4fb6-a05c-bce70bac9342-metrics-certs\") pod \"maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2\" (UID: \"a002f857-7a6e-4fb6-a05c-bce70bac9342\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2"
Oct 30 01:15:57.016076 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:57.016052    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a002f857-7a6e-4fb6-a05c-bce70bac9342-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2\" (UID: \"a002f857-7a6e-4fb6-a05c-bce70bac9342\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2"
Oct 30 01:15:57.024310 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:57.024277    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a002f857-7a6e-4fb6-a05c-bce70bac9342-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2\" (UID: \"a002f857-7a6e-4fb6-a05c-bce70bac9342\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2"
Oct 30 01:15:57.048803 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:57.047092    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a002f857-7a6e-4fb6-a05c-bce70bac9342-metrics-certs\") pod \"maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2\" (UID: \"a002f857-7a6e-4fb6-a05c-bce70bac9342\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2"
Oct 30 01:15:57.059465 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:57.059428    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a002f857-7a6e-4fb6-a05c-bce70bac9342-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2\" (UID: \"a002f857-7a6e-4fb6-a05c-bce70bac9342\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2"
Oct 30 01:15:57.061210 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:57.061182    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trphh\" (UniqueName: \"kubernetes.io/projected/a002f857-7a6e-4fb6-a05c-bce70bac9342-kube-api-access-trphh\") pod \"maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2\" (UID: \"a002f857-7a6e-4fb6-a05c-bce70bac9342\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2"
Oct 30 01:15:57.343957 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:57.343878    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2"
Oct 30 01:15:57.650280 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:15:57.650248    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda002f857_7a6e_4fb6_a05c_bce70bac9342.slice/crio-1f6ccd7c8f476269e7a3f35826bba510674c50d62d9f2479c2c57b140a8619f8 WatchSource:0}: Error finding container 1f6ccd7c8f476269e7a3f35826bba510674c50d62d9f2479c2c57b140a8619f8: Status 404 returned error can't find the container with id 1f6ccd7c8f476269e7a3f35826bba510674c50d62d9f2479c2c57b140a8619f8
Oct 30 01:15:57.652859 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:57.652834    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2"]
Oct 30 01:15:57.820565 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:57.820531    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2" event={"ID":"a002f857-7a6e-4fb6-a05c-bce70bac9342","Type":"ContainerStarted","Data":"1f6ccd7c8f476269e7a3f35826bba510674c50d62d9f2479c2c57b140a8619f8"}
Oct 30 01:15:58.935796 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:58.935732    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2" event={"ID":"a002f857-7a6e-4fb6-a05c-bce70bac9342","Type":"ContainerStarted","Data":"88802c51b03f92250f16d66397ea6fbe94f697e6e44b3ae4ab28e29b552d4e9f"}
Oct 30 01:15:58.936202 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:58.936062    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2"
Oct 30 01:15:58.988187 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:15:58.988129    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2" podStartSLOduration=2.6388163970000003 podStartE2EDuration="2.988113688s" totalImagesPullingTime="349.297291ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:15:58.983830153 +0000 UTC m=+2649.371506542" watchObservedRunningTime="2026-10-30 01:15:58.988113688 +0000 UTC m=+2649.375790076"
Oct 30 01:16:04.194626 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:04.194526    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"]
Oct 30 01:16:04.948274 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:04.948246    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2"
Oct 30 01:16:19.056400 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:19.056365    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"]
Oct 30 01:16:19.056992 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:19.056636    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh" podUID="2535b52d-1db6-4f2b-9912-4d9316aaa69d" containerName="istio-proxy" containerID="cri-o://e0e85a64e815237d97347aa41b31af9d929c4631a949e711b9e20a9acde2dbeb" gracePeriod=30
Oct 30 01:16:19.218819 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:19.218758    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g" podUID="fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d" containerName="main" containerID="cri-o://33dec75db61ea84de9262e5b1383f14e3c0c5b7df0fbadc5c454a6ead4476c9c" gracePeriod=45
Oct 30 01:16:19.508046 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:19.508016    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"
Oct 30 01:16:19.624253 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:19.624216    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-tls-certs\" (UniqueName: \"kubernetes.io/secret/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-tls-certs\") pod \"fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d\" (UID: \"fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d\") "
Oct 30 01:16:19.624459 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:19.624378    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-model-cache\") pod \"fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d\" (UID: \"fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d\") "
Oct 30 01:16:19.624681 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:19.624653    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-model-cache" pod "fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d" (UID: "fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:19.624998 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:19.624792    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-tmp-dir\") pod \"fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d\" (UID: \"fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d\") "
Oct 30 01:16:19.625126 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:19.625034    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-dshm\") pod \"fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d\" (UID: \"fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d\") "
Oct 30 01:16:19.625202 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:19.625150    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-home\" (UniqueName: \"kubernetes.io/empty-dir/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-home\") pod \"fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d\" (UID: \"fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d\") "
Oct 30 01:16:19.626066 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:19.625240    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-kube-api-access-67zpt\" (UniqueName: \"kubernetes.io/projected/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-kube-api-access-67zpt\") pod \"fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d\" (UID: \"fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d\") "
Oct 30 01:16:19.626066 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:19.625841    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-home" pod "fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d" (UID: "fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:19.626066 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:19.626004    4393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-home\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:16:19.626066 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:19.626023    4393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-model-cache\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:16:19.626468 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:19.626437    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-tmp-dir" pod "fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d" (UID: "fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:19.628573 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:19.628518    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-tls-certs" pod "fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d" (UID: "fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:16:19.633951 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:19.633927    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-dshm" pod "fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d" (UID: "fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:19.634043 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:19.633978    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-kube-api-access-67zpt" pod "fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d" (UID: "fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d"). InnerVolumeSpecName "kube-api-access-67zpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:19.727726 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:19.727692    4393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-tmp-dir\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:16:19.727998 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:19.727980    4393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-dshm\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:16:19.728113 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:19.728100    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67zpt\" (UniqueName: \"kubernetes.io/projected/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-kube-api-access-67zpt\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:16:19.728200 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:19.728187    4393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d-tls-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:16:20.192221 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:20.192185    4393 generic.go:356] "Generic (PLEG): container finished" podID="fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d" containerID="33dec75db61ea84de9262e5b1383f14e3c0c5b7df0fbadc5c454a6ead4476c9c" exitCode=0
Oct 30 01:16:20.192712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:20.192233    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"
Oct 30 01:16:20.192712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:20.192264    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g" event={"ID":"fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d","Type":"ContainerDied","Data":"33dec75db61ea84de9262e5b1383f14e3c0c5b7df0fbadc5c454a6ead4476c9c"}
Oct 30 01:16:20.192712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:20.192309    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g" event={"ID":"fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d","Type":"ContainerDied","Data":"6bfb6799e0c65bfd1f02da4231da4507ba02e5c8706e0ace81259a09e48be72d"}
Oct 30 01:16:20.192712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:20.192327    4393 scope.go:122] "RemoveContainer" containerID="33dec75db61ea84de9262e5b1383f14e3c0c5b7df0fbadc5c454a6ead4476c9c"
Oct 30 01:16:20.203396 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:20.203371    4393 scope.go:122] "RemoveContainer" containerID="33dec75db61ea84de9262e5b1383f14e3c0c5b7df0fbadc5c454a6ead4476c9c"
Oct 30 01:16:20.203669 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:16:20.203649    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33dec75db61ea84de9262e5b1383f14e3c0c5b7df0fbadc5c454a6ead4476c9c\": container with ID starting with 33dec75db61ea84de9262e5b1383f14e3c0c5b7df0fbadc5c454a6ead4476c9c not found: ID does not exist" containerID="33dec75db61ea84de9262e5b1383f14e3c0c5b7df0fbadc5c454a6ead4476c9c"
Oct 30 01:16:20.203718 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:20.203677    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33dec75db61ea84de9262e5b1383f14e3c0c5b7df0fbadc5c454a6ead4476c9c"} err="failed to get container status \"33dec75db61ea84de9262e5b1383f14e3c0c5b7df0fbadc5c454a6ead4476c9c\": rpc error: code = NotFound desc = could not find container \"33dec75db61ea84de9262e5b1383f14e3c0c5b7df0fbadc5c454a6ead4476c9c\": container with ID starting with 33dec75db61ea84de9262e5b1383f14e3c0c5b7df0fbadc5c454a6ead4476c9c not found: ID does not exist"
Oct 30 01:16:20.244641 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:20.244614    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"]
Oct 30 01:16:20.256920 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:20.256894    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-52650f/distinct-2-w0-52650f-kserve-746fdc9dcd-whp2g"]
Oct 30 01:16:21.429055 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:21.429022    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc"]
Oct 30 01:16:21.429571 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:21.429257    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc" podUID="77782b40-06f9-4ba1-8c0b-fc787ee19a9b" containerName="maas-api" containerID="cri-o://0f088797ac0aaa0aa87dbc96708e36922ccedfd7ed6f5a9554d9fc01b04af4dd" gracePeriod=30
Oct 30 01:16:21.784409 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:21.784367    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d" path="/var/lib/kubelet/pods/fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d/volumes"
Oct 30 01:16:21.815553 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:21.815524    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc"
Oct 30 01:16:22.015856 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:21.993932    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77782b40-06f9-4ba1-8c0b-fc787ee19a9b-kube-api-access-9m86x\" (UniqueName: \"kubernetes.io/projected/77782b40-06f9-4ba1-8c0b-fc787ee19a9b-kube-api-access-9m86x\") pod \"77782b40-06f9-4ba1-8c0b-fc787ee19a9b\" (UID: \"77782b40-06f9-4ba1-8c0b-fc787ee19a9b\") "
Oct 30 01:16:22.015856 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:21.994003    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77782b40-06f9-4ba1-8c0b-fc787ee19a9b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/77782b40-06f9-4ba1-8c0b-fc787ee19a9b-metrics-certs\") pod \"77782b40-06f9-4ba1-8c0b-fc787ee19a9b\" (UID: \"77782b40-06f9-4ba1-8c0b-fc787ee19a9b\") "
Oct 30 01:16:22.015856 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:21.994089    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77782b40-06f9-4ba1-8c0b-fc787ee19a9b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77782b40-06f9-4ba1-8c0b-fc787ee19a9b-maas-api-tls\") pod \"77782b40-06f9-4ba1-8c0b-fc787ee19a9b\" (UID: \"77782b40-06f9-4ba1-8c0b-fc787ee19a9b\") "
Oct 30 01:16:22.015856 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:21.994149    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77782b40-06f9-4ba1-8c0b-fc787ee19a9b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77782b40-06f9-4ba1-8c0b-fc787ee19a9b-openshift-service-ca\") pod \"77782b40-06f9-4ba1-8c0b-fc787ee19a9b\" (UID: \"77782b40-06f9-4ba1-8c0b-fc787ee19a9b\") "
Oct 30 01:16:22.015856 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:21.995384    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77782b40-06f9-4ba1-8c0b-fc787ee19a9b-openshift-service-ca" pod "77782b40-06f9-4ba1-8c0b-fc787ee19a9b" (UID: "77782b40-06f9-4ba1-8c0b-fc787ee19a9b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:16:22.015856 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:22.000156    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77782b40-06f9-4ba1-8c0b-fc787ee19a9b-metrics-certs" pod "77782b40-06f9-4ba1-8c0b-fc787ee19a9b" (UID: "77782b40-06f9-4ba1-8c0b-fc787ee19a9b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:16:22.044802 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:22.040282    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77782b40-06f9-4ba1-8c0b-fc787ee19a9b-kube-api-access-9m86x" pod "77782b40-06f9-4ba1-8c0b-fc787ee19a9b" (UID: "77782b40-06f9-4ba1-8c0b-fc787ee19a9b"). InnerVolumeSpecName "kube-api-access-9m86x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:22.044802 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:22.040431    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77782b40-06f9-4ba1-8c0b-fc787ee19a9b-maas-api-tls" pod "77782b40-06f9-4ba1-8c0b-fc787ee19a9b" (UID: "77782b40-06f9-4ba1-8c0b-fc787ee19a9b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:16:22.096626 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:22.096590    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m86x\" (UniqueName: \"kubernetes.io/projected/77782b40-06f9-4ba1-8c0b-fc787ee19a9b-kube-api-access-9m86x\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:16:22.096626 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:22.096627    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77782b40-06f9-4ba1-8c0b-fc787ee19a9b-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:16:22.096917 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:22.096641    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77782b40-06f9-4ba1-8c0b-fc787ee19a9b-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:16:22.096917 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:22.096654    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77782b40-06f9-4ba1-8c0b-fc787ee19a9b-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:16:22.210636 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:22.210606    4393 generic.go:356] "Generic (PLEG): container finished" podID="77782b40-06f9-4ba1-8c0b-fc787ee19a9b" containerID="0f088797ac0aaa0aa87dbc96708e36922ccedfd7ed6f5a9554d9fc01b04af4dd" exitCode=0
Oct 30 01:16:22.210849 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:22.210654    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc"
Oct 30 01:16:22.210849 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:22.210701    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc" event={"ID":"77782b40-06f9-4ba1-8c0b-fc787ee19a9b","Type":"ContainerDied","Data":"0f088797ac0aaa0aa87dbc96708e36922ccedfd7ed6f5a9554d9fc01b04af4dd"}
Oct 30 01:16:22.210849 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:22.210755    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc" event={"ID":"77782b40-06f9-4ba1-8c0b-fc787ee19a9b","Type":"ContainerDied","Data":"34b0342bd32f7f445d0ee9e420bd12baa51e67ff291f686a33d5c7687373742e"}
Oct 30 01:16:22.210849 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:22.210817    4393 scope.go:122] "RemoveContainer" containerID="0f088797ac0aaa0aa87dbc96708e36922ccedfd7ed6f5a9554d9fc01b04af4dd"
Oct 30 01:16:22.226146 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:22.226128    4393 scope.go:122] "RemoveContainer" containerID="0f088797ac0aaa0aa87dbc96708e36922ccedfd7ed6f5a9554d9fc01b04af4dd"
Oct 30 01:16:22.226609 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:16:22.226588    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f088797ac0aaa0aa87dbc96708e36922ccedfd7ed6f5a9554d9fc01b04af4dd\": container with ID starting with 0f088797ac0aaa0aa87dbc96708e36922ccedfd7ed6f5a9554d9fc01b04af4dd not found: ID does not exist" containerID="0f088797ac0aaa0aa87dbc96708e36922ccedfd7ed6f5a9554d9fc01b04af4dd"
Oct 30 01:16:22.226692 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:22.226616    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f088797ac0aaa0aa87dbc96708e36922ccedfd7ed6f5a9554d9fc01b04af4dd"} err="failed to get container status \"0f088797ac0aaa0aa87dbc96708e36922ccedfd7ed6f5a9554d9fc01b04af4dd\": rpc error: code = NotFound desc = could not find container \"0f088797ac0aaa0aa87dbc96708e36922ccedfd7ed6f5a9554d9fc01b04af4dd\": container with ID starting with 0f088797ac0aaa0aa87dbc96708e36922ccedfd7ed6f5a9554d9fc01b04af4dd not found: ID does not exist"
Oct 30 01:16:22.263835 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:22.263803    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc"]
Oct 30 01:16:22.268741 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:22.268677    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-48df96-59d7ff44f-b2cwc"]
Oct 30 01:16:23.733128 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:23.733096    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77782b40-06f9-4ba1-8c0b-fc787ee19a9b" path="/var/lib/kubelet/pods/77782b40-06f9-4ba1-8c0b-fc787ee19a9b/volumes"
Oct 30 01:16:24.258851 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:24.258801    4393 generic.go:356] "Generic (PLEG): container finished" podID="2535b52d-1db6-4f2b-9912-4d9316aaa69d" containerID="e0e85a64e815237d97347aa41b31af9d929c4631a949e711b9e20a9acde2dbeb" exitCode=0
Oct 30 01:16:24.259044 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:24.258988    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh" event={"ID":"2535b52d-1db6-4f2b-9912-4d9316aaa69d","Type":"ContainerDied","Data":"e0e85a64e815237d97347aa41b31af9d929c4631a949e711b9e20a9acde2dbeb"}
Oct 30 01:16:24.390578 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:24.390544    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"
Oct 30 01:16:24.448270 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:24.448239    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2535b52d-1db6-4f2b-9912-4d9316aaa69d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2535b52d-1db6-4f2b-9912-4d9316aaa69d-credential-socket\") pod \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\" (UID: \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\") "
Oct 30 01:16:24.448476 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:24.448309    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2535b52d-1db6-4f2b-9912-4d9316aaa69d-kube-api-access-qbbxc\" (UniqueName: \"kubernetes.io/projected/2535b52d-1db6-4f2b-9912-4d9316aaa69d-kube-api-access-qbbxc\") pod \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\" (UID: \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\") "
Oct 30 01:16:24.448476 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:24.448353    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2535b52d-1db6-4f2b-9912-4d9316aaa69d-istio-token\" (UniqueName: \"kubernetes.io/projected/2535b52d-1db6-4f2b-9912-4d9316aaa69d-istio-token\") pod \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\" (UID: \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\") "
Oct 30 01:16:24.448476 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:24.448384    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2535b52d-1db6-4f2b-9912-4d9316aaa69d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2535b52d-1db6-4f2b-9912-4d9316aaa69d-istiod-ca-cert\") pod \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\" (UID: \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\") "
Oct 30 01:16:24.448476 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:24.448448    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2535b52d-1db6-4f2b-9912-4d9316aaa69d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2535b52d-1db6-4f2b-9912-4d9316aaa69d-istio-data\") pod \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\" (UID: \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\") "
Oct 30 01:16:24.448681 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:24.448488    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2535b52d-1db6-4f2b-9912-4d9316aaa69d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2535b52d-1db6-4f2b-9912-4d9316aaa69d-istio-envoy\") pod \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\" (UID: \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\") "
Oct 30 01:16:24.448681 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:24.448555    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2535b52d-1db6-4f2b-9912-4d9316aaa69d-credential-socket" pod "2535b52d-1db6-4f2b-9912-4d9316aaa69d" (UID: "2535b52d-1db6-4f2b-9912-4d9316aaa69d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:24.448830 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:24.448718    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2535b52d-1db6-4f2b-9912-4d9316aaa69d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2535b52d-1db6-4f2b-9912-4d9316aaa69d-istio-podinfo\") pod \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\" (UID: \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\") "
Oct 30 01:16:24.448830 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:24.448735    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2535b52d-1db6-4f2b-9912-4d9316aaa69d-istio-data" pod "2535b52d-1db6-4f2b-9912-4d9316aaa69d" (UID: "2535b52d-1db6-4f2b-9912-4d9316aaa69d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:24.448830 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:24.448816    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2535b52d-1db6-4f2b-9912-4d9316aaa69d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2535b52d-1db6-4f2b-9912-4d9316aaa69d-workload-certs\") pod \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\" (UID: \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\") "
Oct 30 01:16:24.449022 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:24.448849    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2535b52d-1db6-4f2b-9912-4d9316aaa69d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2535b52d-1db6-4f2b-9912-4d9316aaa69d-workload-socket\") pod \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\" (UID: \"2535b52d-1db6-4f2b-9912-4d9316aaa69d\") "
Oct 30 01:16:24.449022 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:24.448883    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2535b52d-1db6-4f2b-9912-4d9316aaa69d-istiod-ca-cert" pod "2535b52d-1db6-4f2b-9912-4d9316aaa69d" (UID: "2535b52d-1db6-4f2b-9912-4d9316aaa69d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:16:24.449325 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:24.449299    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2535b52d-1db6-4f2b-9912-4d9316aaa69d-workload-certs" pod "2535b52d-1db6-4f2b-9912-4d9316aaa69d" (UID: "2535b52d-1db6-4f2b-9912-4d9316aaa69d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:24.449442 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:24.449387    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2535b52d-1db6-4f2b-9912-4d9316aaa69d-workload-socket" pod "2535b52d-1db6-4f2b-9912-4d9316aaa69d" (UID: "2535b52d-1db6-4f2b-9912-4d9316aaa69d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:24.450067 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:24.450043    4393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2535b52d-1db6-4f2b-9912-4d9316aaa69d-workload-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:16:24.450181 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:24.450073    4393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2535b52d-1db6-4f2b-9912-4d9316aaa69d-workload-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:16:24.450181 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:24.450088    4393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2535b52d-1db6-4f2b-9912-4d9316aaa69d-credential-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:16:24.450181 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:24.450104    4393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2535b52d-1db6-4f2b-9912-4d9316aaa69d-istiod-ca-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:16:24.450181 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:24.450119    4393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2535b52d-1db6-4f2b-9912-4d9316aaa69d-istio-data\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:16:24.452912 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:24.451361    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2535b52d-1db6-4f2b-9912-4d9316aaa69d-istio-token" pod "2535b52d-1db6-4f2b-9912-4d9316aaa69d" (UID: "2535b52d-1db6-4f2b-9912-4d9316aaa69d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:24.452912 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:24.452024    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2535b52d-1db6-4f2b-9912-4d9316aaa69d-istio-podinfo" pod "2535b52d-1db6-4f2b-9912-4d9316aaa69d" (UID: "2535b52d-1db6-4f2b-9912-4d9316aaa69d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:16:24.452912 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:24.452056    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2535b52d-1db6-4f2b-9912-4d9316aaa69d-istio-envoy" pod "2535b52d-1db6-4f2b-9912-4d9316aaa69d" (UID: "2535b52d-1db6-4f2b-9912-4d9316aaa69d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:16:24.457131 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:24.457109    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2535b52d-1db6-4f2b-9912-4d9316aaa69d-kube-api-access-qbbxc" pod "2535b52d-1db6-4f2b-9912-4d9316aaa69d" (UID: "2535b52d-1db6-4f2b-9912-4d9316aaa69d"). InnerVolumeSpecName "kube-api-access-qbbxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:16:24.552073 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:24.552041    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2535b52d-1db6-4f2b-9912-4d9316aaa69d-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:16:24.552073 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:24.552068    4393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2535b52d-1db6-4f2b-9912-4d9316aaa69d-istio-envoy\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:16:24.552073 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:24.552077    4393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2535b52d-1db6-4f2b-9912-4d9316aaa69d-istio-podinfo\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:16:24.552299 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:24.552086    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbbxc\" (UniqueName: \"kubernetes.io/projected/2535b52d-1db6-4f2b-9912-4d9316aaa69d-kube-api-access-qbbxc\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:16:25.270846 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:25.270801    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"
Oct 30 01:16:25.270846 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:25.270832    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh" event={"ID":"2535b52d-1db6-4f2b-9912-4d9316aaa69d","Type":"ContainerDied","Data":"7ce993e3367534b37827c5996c9db14d6f50b6b239f7207d10836228bbaa8d85"}
Oct 30 01:16:25.271340 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:25.270896    4393 scope.go:122] "RemoveContainer" containerID="e0e85a64e815237d97347aa41b31af9d929c4631a949e711b9e20a9acde2dbeb"
Oct 30 01:16:25.338083 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:25.338039    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"]
Oct 30 01:16:25.346942 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:25.346907    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-48df96-openshift-default-5c8d99d755-h65wh"]
Oct 30 01:16:25.800444 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:25.796290    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2535b52d-1db6-4f2b-9912-4d9316aaa69d" path="/var/lib/kubelet/pods/2535b52d-1db6-4f2b-9912-4d9316aaa69d/volumes"
Oct 30 01:16:26.512130 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:16:26.512089    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:16:54.669708 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:54.669659    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k"]
Oct 30 01:16:54.670705 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:54.670683    4393 cpu_manager.go:414] "containerMap: removing container" podUID="fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d" containerName="main"
Oct 30 01:16:54.670705 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:54.670706    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d" containerName="main"
Oct 30 01:16:54.670864 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:54.670731    4393 cpu_manager.go:414] "containerMap: removing container" podUID="2535b52d-1db6-4f2b-9912-4d9316aaa69d" containerName="istio-proxy"
Oct 30 01:16:54.670864 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:54.670737    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2535b52d-1db6-4f2b-9912-4d9316aaa69d" containerName="istio-proxy"
Oct 30 01:16:54.670864 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:54.670791    4393 cpu_manager.go:414] "containerMap: removing container" podUID="77782b40-06f9-4ba1-8c0b-fc787ee19a9b" containerName="maas-api"
Oct 30 01:16:54.670864 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:54.670799    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77782b40-06f9-4ba1-8c0b-fc787ee19a9b" containerName="maas-api"
Oct 30 01:16:54.671042 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:54.670944    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="77782b40-06f9-4ba1-8c0b-fc787ee19a9b" containerName="maas-api"
Oct 30 01:16:54.671042 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:54.670956    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc843ff2-b1a1-4e3c-a7ee-2b349a074b4d" containerName="main"
Oct 30 01:16:54.671042 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:54.670964    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="2535b52d-1db6-4f2b-9912-4d9316aaa69d" containerName="istio-proxy"
Oct 30 01:16:54.676198 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:54.676176    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k"
Oct 30 01:16:54.699515 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:54.699483    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k"]
Oct 30 01:16:54.704575 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:54.704548    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a7ca1f\"/\"default-dockercfg-g48sw\""
Oct 30 01:16:54.704727 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:54.704583    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a7ca1f\"/\"kube-root-ca.crt\""
Oct 30 01:16:54.704727 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:54.704609    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a7ca1f\"/\"ipp-route-a7ca1f-kserve-self-signed-certs\""
Oct 30 01:16:54.704727 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:54.704648    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a7ca1f\"/\"openshift-service-ca.crt\""
Oct 30 01:16:54.817700 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:54.817657    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f22d339c-0c05-4da1-90d1-062f85e597de-model-cache\") pod \"ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k\" (UID: \"f22d339c-0c05-4da1-90d1-062f85e597de\") " pod="ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k"
Oct 30 01:16:54.817927 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:54.817743    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f22d339c-0c05-4da1-90d1-062f85e597de-home\") pod \"ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k\" (UID: \"f22d339c-0c05-4da1-90d1-062f85e597de\") " pod="ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k"
Oct 30 01:16:54.817927 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:54.817813    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqlk2\" (UniqueName: \"kubernetes.io/projected/f22d339c-0c05-4da1-90d1-062f85e597de-kube-api-access-jqlk2\") pod \"ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k\" (UID: \"f22d339c-0c05-4da1-90d1-062f85e597de\") " pod="ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k"
Oct 30 01:16:54.818011 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:54.817996    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f22d339c-0c05-4da1-90d1-062f85e597de-dshm\") pod \"ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k\" (UID: \"f22d339c-0c05-4da1-90d1-062f85e597de\") " pod="ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k"
Oct 30 01:16:54.818091 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:54.818073    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f22d339c-0c05-4da1-90d1-062f85e597de-tls-certs\") pod \"ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k\" (UID: \"f22d339c-0c05-4da1-90d1-062f85e597de\") " pod="ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k"
Oct 30 01:16:54.818131 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:54.818115    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f22d339c-0c05-4da1-90d1-062f85e597de-tmp-dir\") pod \"ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k\" (UID: \"f22d339c-0c05-4da1-90d1-062f85e597de\") " pod="ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k"
Oct 30 01:16:54.940895 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:54.940797    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f22d339c-0c05-4da1-90d1-062f85e597de-tls-certs\") pod \"ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k\" (UID: \"f22d339c-0c05-4da1-90d1-062f85e597de\") " pod="ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k"
Oct 30 01:16:54.941172 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:54.941148    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f22d339c-0c05-4da1-90d1-062f85e597de-tmp-dir\") pod \"ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k\" (UID: \"f22d339c-0c05-4da1-90d1-062f85e597de\") " pod="ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k"
Oct 30 01:16:54.949901 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:54.949864    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f22d339c-0c05-4da1-90d1-062f85e597de-model-cache\") pod \"ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k\" (UID: \"f22d339c-0c05-4da1-90d1-062f85e597de\") " pod="ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k"
Oct 30 01:16:54.950145 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:54.950124    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f22d339c-0c05-4da1-90d1-062f85e597de-home\") pod \"ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k\" (UID: \"f22d339c-0c05-4da1-90d1-062f85e597de\") " pod="ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k"
Oct 30 01:16:54.950290 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:54.950274    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqlk2\" (UniqueName: \"kubernetes.io/projected/f22d339c-0c05-4da1-90d1-062f85e597de-kube-api-access-jqlk2\") pod \"ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k\" (UID: \"f22d339c-0c05-4da1-90d1-062f85e597de\") " pod="ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k"
Oct 30 01:16:54.950605 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:54.950581    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f22d339c-0c05-4da1-90d1-062f85e597de-dshm\") pod \"ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k\" (UID: \"f22d339c-0c05-4da1-90d1-062f85e597de\") " pod="ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k"
Oct 30 01:16:54.957800 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:54.957175    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f22d339c-0c05-4da1-90d1-062f85e597de-model-cache\") pod \"ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k\" (UID: \"f22d339c-0c05-4da1-90d1-062f85e597de\") " pod="ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k"
Oct 30 01:16:54.957922 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:54.957869    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f22d339c-0c05-4da1-90d1-062f85e597de-tmp-dir\") pod \"ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k\" (UID: \"f22d339c-0c05-4da1-90d1-062f85e597de\") " pod="ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k"
Oct 30 01:16:54.969805 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:54.958115    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f22d339c-0c05-4da1-90d1-062f85e597de-home\") pod \"ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k\" (UID: \"f22d339c-0c05-4da1-90d1-062f85e597de\") " pod="ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k"
Oct 30 01:16:54.983872 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:54.982014    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f22d339c-0c05-4da1-90d1-062f85e597de-tls-certs\") pod \"ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k\" (UID: \"f22d339c-0c05-4da1-90d1-062f85e597de\") " pod="ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k"
Oct 30 01:16:54.987086 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:54.987054    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f22d339c-0c05-4da1-90d1-062f85e597de-dshm\") pod \"ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k\" (UID: \"f22d339c-0c05-4da1-90d1-062f85e597de\") " pod="ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k"
Oct 30 01:16:55.005807 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:55.004489    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqlk2\" (UniqueName: \"kubernetes.io/projected/f22d339c-0c05-4da1-90d1-062f85e597de-kube-api-access-jqlk2\") pod \"ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k\" (UID: \"f22d339c-0c05-4da1-90d1-062f85e597de\") " pod="ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k"
Oct 30 01:16:55.301838 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:55.301725    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k"
Oct 30 01:16:55.613348 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:55.613316    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k"]
Oct 30 01:16:55.613493 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:16:55.613313    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf22d339c_0c05_4da1_90d1_062f85e597de.slice/crio-4c2b02e240321f14e6aa66a042f06a8ce4c7ac95d3cd988e928558d2b89f2986 WatchSource:0}: Error finding container 4c2b02e240321f14e6aa66a042f06a8ce4c7ac95d3cd988e928558d2b89f2986: Status 404 returned error can't find the container with id 4c2b02e240321f14e6aa66a042f06a8ce4c7ac95d3cd988e928558d2b89f2986
Oct 30 01:16:56.512769 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:16:56.512732    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:16:56.548171 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:56.548131    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k" event={"ID":"f22d339c-0c05-4da1-90d1-062f85e597de","Type":"ContainerStarted","Data":"91be60d9888388c45963513bae1c5cd9e0a4366764daca484ba0b8eb67673608"}
Oct 30 01:16:56.548346 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:56.548182    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k" event={"ID":"f22d339c-0c05-4da1-90d1-062f85e597de","Type":"ContainerStarted","Data":"4c2b02e240321f14e6aa66a042f06a8ce4c7ac95d3cd988e928558d2b89f2986"}
Oct 30 01:16:56.590310 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:56.590260    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k" podStartSLOduration=2.590244929 podStartE2EDuration="2.590244929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:56.588556468 +0000 UTC m=+2706.976232856" watchObservedRunningTime="2026-10-30 01:16:56.590244929 +0000 UTC m=+2706.977921318"
Oct 30 01:16:57.302360 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:16:57.302317    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k"
Oct 30 01:17:05.302674 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:05.302634    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k"
Oct 30 01:17:05.315897 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:05.315870    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k"
Oct 30 01:17:05.630682 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:05.630655    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k"
Oct 30 01:17:11.322666 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:11.322634    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc"]
Oct 30 01:17:11.327694 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:11.327674    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc"
Oct 30 01:17:11.348165 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:11.348139    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc"]
Oct 30 01:17:11.353935 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:11.353908    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-9a8001-serving-cert\""
Oct 30 01:17:11.413480 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:11.413448    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25aa52b6-2bb8-457c-9708-d4147aed2544-metrics-certs\") pod \"maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc\" (UID: \"25aa52b6-2bb8-457c-9708-d4147aed2544\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc"
Oct 30 01:17:11.413665 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:11.413582    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlmnh\" (UniqueName: \"kubernetes.io/projected/25aa52b6-2bb8-457c-9708-d4147aed2544-kube-api-access-tlmnh\") pod \"maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc\" (UID: \"25aa52b6-2bb8-457c-9708-d4147aed2544\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc"
Oct 30 01:17:11.413894 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:11.413870    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25aa52b6-2bb8-457c-9708-d4147aed2544-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc\" (UID: \"25aa52b6-2bb8-457c-9708-d4147aed2544\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc"
Oct 30 01:17:11.413980 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:11.413920    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25aa52b6-2bb8-457c-9708-d4147aed2544-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc\" (UID: \"25aa52b6-2bb8-457c-9708-d4147aed2544\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc"
Oct 30 01:17:11.515503 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:11.515461    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25aa52b6-2bb8-457c-9708-d4147aed2544-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc\" (UID: \"25aa52b6-2bb8-457c-9708-d4147aed2544\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc"
Oct 30 01:17:11.515713 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:11.515526    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25aa52b6-2bb8-457c-9708-d4147aed2544-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc\" (UID: \"25aa52b6-2bb8-457c-9708-d4147aed2544\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc"
Oct 30 01:17:11.515713 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:11.515595    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25aa52b6-2bb8-457c-9708-d4147aed2544-metrics-certs\") pod \"maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc\" (UID: \"25aa52b6-2bb8-457c-9708-d4147aed2544\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc"
Oct 30 01:17:11.515713 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:17:11.515618    4393 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-9a8001-serving-cert: secret "maas-api-e2e-ipp-b-9a8001-serving-cert" not found
Oct 30 01:17:11.515713 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:11.515675    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlmnh\" (UniqueName: \"kubernetes.io/projected/25aa52b6-2bb8-457c-9708-d4147aed2544-kube-api-access-tlmnh\") pod \"maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc\" (UID: \"25aa52b6-2bb8-457c-9708-d4147aed2544\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc"
Oct 30 01:17:11.515713 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:17:11.515699    4393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25aa52b6-2bb8-457c-9708-d4147aed2544-maas-api-tls podName:25aa52b6-2bb8-457c-9708-d4147aed2544 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:12.01567982 +0000 UTC m=+2722.403356202 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/25aa52b6-2bb8-457c-9708-d4147aed2544-maas-api-tls") pod "maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc" (UID: "25aa52b6-2bb8-457c-9708-d4147aed2544") : secret "maas-api-e2e-ipp-b-9a8001-serving-cert" not found
Oct 30 01:17:11.516446 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:11.516421    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25aa52b6-2bb8-457c-9708-d4147aed2544-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc\" (UID: \"25aa52b6-2bb8-457c-9708-d4147aed2544\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc"
Oct 30 01:17:11.539852 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:11.526536    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25aa52b6-2bb8-457c-9708-d4147aed2544-metrics-certs\") pod \"maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc\" (UID: \"25aa52b6-2bb8-457c-9708-d4147aed2544\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc"
Oct 30 01:17:11.573688 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:11.573605    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlmnh\" (UniqueName: \"kubernetes.io/projected/25aa52b6-2bb8-457c-9708-d4147aed2544-kube-api-access-tlmnh\") pod \"maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc\" (UID: \"25aa52b6-2bb8-457c-9708-d4147aed2544\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc"
Oct 30 01:17:12.023143 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:12.023099    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25aa52b6-2bb8-457c-9708-d4147aed2544-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc\" (UID: \"25aa52b6-2bb8-457c-9708-d4147aed2544\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc"
Oct 30 01:17:12.025637 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:12.025616    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25aa52b6-2bb8-457c-9708-d4147aed2544-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc\" (UID: \"25aa52b6-2bb8-457c-9708-d4147aed2544\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc"
Oct 30 01:17:12.244213 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:12.244174    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc"
Oct 30 01:17:12.573562 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:12.573533    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc"]
Oct 30 01:17:12.573938 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:17:12.573693    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25aa52b6_2bb8_457c_9708_d4147aed2544.slice/crio-1cd5e196754ac806cf966fbab8461db54b35ce8dddd0aed1ff8299dd0ffb0705 WatchSource:0}: Error finding container 1cd5e196754ac806cf966fbab8461db54b35ce8dddd0aed1ff8299dd0ffb0705: Status 404 returned error can't find the container with id 1cd5e196754ac806cf966fbab8461db54b35ce8dddd0aed1ff8299dd0ffb0705
Oct 30 01:17:12.575287 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:12.575270    4393 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 01:17:12.719633 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:12.719595    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc" event={"ID":"25aa52b6-2bb8-457c-9708-d4147aed2544","Type":"ContainerStarted","Data":"1cd5e196754ac806cf966fbab8461db54b35ce8dddd0aed1ff8299dd0ffb0705"}
Oct 30 01:17:13.729939 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:13.729906    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc" event={"ID":"25aa52b6-2bb8-457c-9708-d4147aed2544","Type":"ContainerStarted","Data":"b5c6ee3766c230cb0f0f5d64679aeb70c14aaaaf333c1707f6e6830cd3a90f80"}
Oct 30 01:17:13.730349 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:13.729959    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc"
Oct 30 01:17:13.773129 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:13.773079    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc" podStartSLOduration=2.336770155 podStartE2EDuration="2.773064653s" totalImagesPullingTime="436.294498ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:13.771053396 +0000 UTC m=+2724.158729786" watchObservedRunningTime="2026-10-30 01:17:13.773064653 +0000 UTC m=+2724.160741069"
Oct 30 01:17:19.745094 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:19.745062    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc"
Oct 30 01:17:25.483460 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:25.483425    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs"]
Oct 30 01:17:25.483983 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:25.483728    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs" podUID="2f1a4faf-e09e-4483-a249-c80e24f81129" containerName="maas-api" containerID="cri-o://6859efafc906d091e92fdd2992064a28fa3db21f8cf94547b6d3014dc7d950f0" gracePeriod=30
Oct 30 01:17:25.842940 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:25.842757    4393 generic.go:356] "Generic (PLEG): container finished" podID="2f1a4faf-e09e-4483-a249-c80e24f81129" containerID="6859efafc906d091e92fdd2992064a28fa3db21f8cf94547b6d3014dc7d950f0" exitCode=0
Oct 30 01:17:25.842940 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:25.842811    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs" event={"ID":"2f1a4faf-e09e-4483-a249-c80e24f81129","Type":"ContainerDied","Data":"6859efafc906d091e92fdd2992064a28fa3db21f8cf94547b6d3014dc7d950f0"}
Oct 30 01:17:25.869407 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:25.869377    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs"
Oct 30 01:17:25.941804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:25.941082    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1a4faf-e09e-4483-a249-c80e24f81129-kube-api-access-wkt5f\" (UniqueName: \"kubernetes.io/projected/2f1a4faf-e09e-4483-a249-c80e24f81129-kube-api-access-wkt5f\") pod \"2f1a4faf-e09e-4483-a249-c80e24f81129\" (UID: \"2f1a4faf-e09e-4483-a249-c80e24f81129\") "
Oct 30 01:17:25.941804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:25.941360    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f1a4faf-e09e-4483-a249-c80e24f81129-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f1a4faf-e09e-4483-a249-c80e24f81129-maas-api-tls\") pod \"2f1a4faf-e09e-4483-a249-c80e24f81129\" (UID: \"2f1a4faf-e09e-4483-a249-c80e24f81129\") "
Oct 30 01:17:25.941804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:25.941458    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f1a4faf-e09e-4483-a249-c80e24f81129-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1a4faf-e09e-4483-a249-c80e24f81129-openshift-service-ca\") pod \"2f1a4faf-e09e-4483-a249-c80e24f81129\" (UID: \"2f1a4faf-e09e-4483-a249-c80e24f81129\") "
Oct 30 01:17:25.941804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:25.941530    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f1a4faf-e09e-4483-a249-c80e24f81129-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f1a4faf-e09e-4483-a249-c80e24f81129-metrics-certs\") pod \"2f1a4faf-e09e-4483-a249-c80e24f81129\" (UID: \"2f1a4faf-e09e-4483-a249-c80e24f81129\") "
Oct 30 01:17:25.959807 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:25.949921    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f1a4faf-e09e-4483-a249-c80e24f81129-openshift-service-ca" pod "2f1a4faf-e09e-4483-a249-c80e24f81129" (UID: "2f1a4faf-e09e-4483-a249-c80e24f81129"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:17:25.978460 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:25.978423    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f1a4faf-e09e-4483-a249-c80e24f81129-metrics-certs" pod "2f1a4faf-e09e-4483-a249-c80e24f81129" (UID: "2f1a4faf-e09e-4483-a249-c80e24f81129"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:17:25.999806 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:25.998392    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f1a4faf-e09e-4483-a249-c80e24f81129-maas-api-tls" pod "2f1a4faf-e09e-4483-a249-c80e24f81129" (UID: "2f1a4faf-e09e-4483-a249-c80e24f81129"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:17:26.000485 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:26.000453    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1a4faf-e09e-4483-a249-c80e24f81129-kube-api-access-wkt5f" pod "2f1a4faf-e09e-4483-a249-c80e24f81129" (UID: "2f1a4faf-e09e-4483-a249-c80e24f81129"). InnerVolumeSpecName "kube-api-access-wkt5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:26.049803 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:26.049118    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkt5f\" (UniqueName: \"kubernetes.io/projected/2f1a4faf-e09e-4483-a249-c80e24f81129-kube-api-access-wkt5f\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:17:26.049803 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:26.049160    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f1a4faf-e09e-4483-a249-c80e24f81129-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:17:26.049803 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:26.049177    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1a4faf-e09e-4483-a249-c80e24f81129-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:17:26.049803 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:26.049190    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f1a4faf-e09e-4483-a249-c80e24f81129-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:17:26.516211 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:17:26.516177    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:17:26.852966 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:26.852890    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs"
Oct 30 01:17:26.852966 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:26.852916    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs" event={"ID":"2f1a4faf-e09e-4483-a249-c80e24f81129","Type":"ContainerDied","Data":"73cfbf7074cd6004387ce62e6cf815effeab8e41eb65116d54f4807392ec3eec"}
Oct 30 01:17:26.853183 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:26.852979    4393 scope.go:122] "RemoveContainer" containerID="6859efafc906d091e92fdd2992064a28fa3db21f8cf94547b6d3014dc7d950f0"
Oct 30 01:17:26.904697 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:26.904671    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs"]
Oct 30 01:17:26.909186 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:26.909160    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-52650f-74d88f9768-5ddhs"]
Oct 30 01:17:27.726769 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:27.726734    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f1a4faf-e09e-4483-a249-c80e24f81129" path="/var/lib/kubelet/pods/2f1a4faf-e09e-4483-a249-c80e24f81129/volumes"
Oct 30 01:17:29.829126 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:29.829094    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf"]
Oct 30 01:17:29.829948 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:29.829931    4393 cpu_manager.go:414] "containerMap: removing container" podUID="2f1a4faf-e09e-4483-a249-c80e24f81129" containerName="maas-api"
Oct 30 01:17:29.829999 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:29.829950    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1a4faf-e09e-4483-a249-c80e24f81129" containerName="maas-api"
Oct 30 01:17:29.830105 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:29.830096    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1a4faf-e09e-4483-a249-c80e24f81129" containerName="maas-api"
Oct 30 01:17:29.835254 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:29.835235    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf"
Oct 30 01:17:29.854238 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:29.854202    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf"]
Oct 30 01:17:29.860794 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:29.860747    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-812ba2-serving-cert\""
Oct 30 01:17:29.903370 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:29.903333    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f8ce06-4e78-43be-9c07-05c201eb70e8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf\" (UID: \"85f8ce06-4e78-43be-9c07-05c201eb70e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf"
Oct 30 01:17:29.903586 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:29.903385    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f8ce06-4e78-43be-9c07-05c201eb70e8-metrics-certs\") pod \"maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf\" (UID: \"85f8ce06-4e78-43be-9c07-05c201eb70e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf"
Oct 30 01:17:29.903586 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:29.903542    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59jjc\" (UniqueName: \"kubernetes.io/projected/85f8ce06-4e78-43be-9c07-05c201eb70e8-kube-api-access-59jjc\") pod \"maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf\" (UID: \"85f8ce06-4e78-43be-9c07-05c201eb70e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf"
Oct 30 01:17:29.903883 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:29.903710    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f8ce06-4e78-43be-9c07-05c201eb70e8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf\" (UID: \"85f8ce06-4e78-43be-9c07-05c201eb70e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf"
Oct 30 01:17:30.004667 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:30.004636    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f8ce06-4e78-43be-9c07-05c201eb70e8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf\" (UID: \"85f8ce06-4e78-43be-9c07-05c201eb70e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf"
Oct 30 01:17:30.004852 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:30.004681    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f8ce06-4e78-43be-9c07-05c201eb70e8-metrics-certs\") pod \"maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf\" (UID: \"85f8ce06-4e78-43be-9c07-05c201eb70e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf"
Oct 30 01:17:30.004852 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:17:30.004752    4393 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-812ba2-serving-cert: secret "maas-api-e2e-ipp-a-812ba2-serving-cert" not found
Oct 30 01:17:30.004852 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:17:30.004836    4393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85f8ce06-4e78-43be-9c07-05c201eb70e8-maas-api-tls podName:85f8ce06-4e78-43be-9c07-05c201eb70e8 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:30.50481618 +0000 UTC m=+2740.892492551 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/85f8ce06-4e78-43be-9c07-05c201eb70e8-maas-api-tls") pod "maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf" (UID: "85f8ce06-4e78-43be-9c07-05c201eb70e8") : secret "maas-api-e2e-ipp-a-812ba2-serving-cert" not found
Oct 30 01:17:30.005014 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:30.004757    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59jjc\" (UniqueName: \"kubernetes.io/projected/85f8ce06-4e78-43be-9c07-05c201eb70e8-kube-api-access-59jjc\") pod \"maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf\" (UID: \"85f8ce06-4e78-43be-9c07-05c201eb70e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf"
Oct 30 01:17:30.005094 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:30.005066    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f8ce06-4e78-43be-9c07-05c201eb70e8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf\" (UID: \"85f8ce06-4e78-43be-9c07-05c201eb70e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf"
Oct 30 01:17:30.005619 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:30.005590    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f8ce06-4e78-43be-9c07-05c201eb70e8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf\" (UID: \"85f8ce06-4e78-43be-9c07-05c201eb70e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf"
Oct 30 01:17:30.007510 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:30.007489    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f8ce06-4e78-43be-9c07-05c201eb70e8-metrics-certs\") pod \"maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf\" (UID: \"85f8ce06-4e78-43be-9c07-05c201eb70e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf"
Oct 30 01:17:30.054214 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:30.051037    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59jjc\" (UniqueName: \"kubernetes.io/projected/85f8ce06-4e78-43be-9c07-05c201eb70e8-kube-api-access-59jjc\") pod \"maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf\" (UID: \"85f8ce06-4e78-43be-9c07-05c201eb70e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf"
Oct 30 01:17:30.530914 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:30.530881    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f8ce06-4e78-43be-9c07-05c201eb70e8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf\" (UID: \"85f8ce06-4e78-43be-9c07-05c201eb70e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf"
Oct 30 01:17:30.533708 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:30.533678    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f8ce06-4e78-43be-9c07-05c201eb70e8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf\" (UID: \"85f8ce06-4e78-43be-9c07-05c201eb70e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf"
Oct 30 01:17:30.752982 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:30.752946    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf"
Oct 30 01:17:30.805861 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:30.805796    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"]
Oct 30 01:17:30.806946 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:30.806362    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc" podUID="123708b7-bba7-4cc1-a2c0-9f3887a742f1" containerName="istio-proxy" containerID="cri-o://2f2fdbbc17929bf3073d94f41ca2218364bf82daa842cdaf50abe3c6e0b1b13d" gracePeriod=30
Oct 30 01:17:31.099568 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:31.099541    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf"]
Oct 30 01:17:31.100197 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:17:31.100171    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f8ce06_4e78_43be_9c07_05c201eb70e8.slice/crio-f71f51023409e8d9ffc53a040bdff3485b4a72d42dc689cdd63b7a3d84a0346c WatchSource:0}: Error finding container f71f51023409e8d9ffc53a040bdff3485b4a72d42dc689cdd63b7a3d84a0346c: Status 404 returned error can't find the container with id f71f51023409e8d9ffc53a040bdff3485b4a72d42dc689cdd63b7a3d84a0346c
Oct 30 01:17:32.022380 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:32.021891    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf" event={"ID":"85f8ce06-4e78-43be-9c07-05c201eb70e8","Type":"ContainerStarted","Data":"1bb1a79e7160656e9c89168c4b69383ec83b6f7ab9e838b2705e67e911a90c60"}
Oct 30 01:17:32.022380 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:32.022006    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf"
Oct 30 01:17:32.022380 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:32.022032    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf" event={"ID":"85f8ce06-4e78-43be-9c07-05c201eb70e8","Type":"ContainerStarted","Data":"f71f51023409e8d9ffc53a040bdff3485b4a72d42dc689cdd63b7a3d84a0346c"}
Oct 30 01:17:32.065058 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:32.065006    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf" podStartSLOduration=2.7517796900000002 podStartE2EDuration="3.064992531s" totalImagesPullingTime="313.212841ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:32.064298909 +0000 UTC m=+2742.451975299" watchObservedRunningTime="2026-10-30 01:17:32.064992531 +0000 UTC m=+2742.452668919"
Oct 30 01:17:33.602621 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:33.602566    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"]
Oct 30 01:17:33.603157 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:33.602872    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x" podUID="d4370a67-835f-4d52-97c4-0419593f7975" containerName="istio-proxy" containerID="cri-o://8691ab65bf40b8deec2745258bb8493da0eccda6f7bb51ffab2ce426d9a409e9" gracePeriod=30
Oct 30 01:17:36.063352 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:36.063318    4393 generic.go:356] "Generic (PLEG): container finished" podID="123708b7-bba7-4cc1-a2c0-9f3887a742f1" containerID="2f2fdbbc17929bf3073d94f41ca2218364bf82daa842cdaf50abe3c6e0b1b13d" exitCode=0
Oct 30 01:17:36.063890 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:36.063372    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc" event={"ID":"123708b7-bba7-4cc1-a2c0-9f3887a742f1","Type":"ContainerDied","Data":"2f2fdbbc17929bf3073d94f41ca2218364bf82daa842cdaf50abe3c6e0b1b13d"}
Oct 30 01:17:36.269016 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:36.268982    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj"]
Oct 30 01:17:36.538393 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:36.538325    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"
Oct 30 01:17:36.584322 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:36.579940    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/123708b7-bba7-4cc1-a2c0-9f3887a742f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/123708b7-bba7-4cc1-a2c0-9f3887a742f1-credential-socket\") pod \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\" (UID: \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\") "
Oct 30 01:17:36.584322 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:36.580028    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/123708b7-bba7-4cc1-a2c0-9f3887a742f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/123708b7-bba7-4cc1-a2c0-9f3887a742f1-istiod-ca-cert\") pod \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\" (UID: \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\") "
Oct 30 01:17:36.584322 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:36.580078    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/123708b7-bba7-4cc1-a2c0-9f3887a742f1-istio-token\" (UniqueName: \"kubernetes.io/projected/123708b7-bba7-4cc1-a2c0-9f3887a742f1-istio-token\") pod \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\" (UID: \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\") "
Oct 30 01:17:36.584322 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:36.580129    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/123708b7-bba7-4cc1-a2c0-9f3887a742f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/123708b7-bba7-4cc1-a2c0-9f3887a742f1-istio-podinfo\") pod \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\" (UID: \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\") "
Oct 30 01:17:36.584322 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:36.580303    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/123708b7-bba7-4cc1-a2c0-9f3887a742f1-kube-api-access-cmf82\" (UniqueName: \"kubernetes.io/projected/123708b7-bba7-4cc1-a2c0-9f3887a742f1-kube-api-access-cmf82\") pod \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\" (UID: \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\") "
Oct 30 01:17:36.584322 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:36.580357    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/123708b7-bba7-4cc1-a2c0-9f3887a742f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/123708b7-bba7-4cc1-a2c0-9f3887a742f1-workload-socket\") pod \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\" (UID: \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\") "
Oct 30 01:17:36.584322 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:36.580403    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/123708b7-bba7-4cc1-a2c0-9f3887a742f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/123708b7-bba7-4cc1-a2c0-9f3887a742f1-istio-data\") pod \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\" (UID: \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\") "
Oct 30 01:17:36.584322 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:36.580518    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/123708b7-bba7-4cc1-a2c0-9f3887a742f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/123708b7-bba7-4cc1-a2c0-9f3887a742f1-workload-certs\") pod \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\" (UID: \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\") "
Oct 30 01:17:36.584322 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:36.580564    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/123708b7-bba7-4cc1-a2c0-9f3887a742f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/123708b7-bba7-4cc1-a2c0-9f3887a742f1-istio-envoy\") pod \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\" (UID: \"123708b7-bba7-4cc1-a2c0-9f3887a742f1\") "
Oct 30 01:17:36.596843 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:36.596796    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123708b7-bba7-4cc1-a2c0-9f3887a742f1-credential-socket" pod "123708b7-bba7-4cc1-a2c0-9f3887a742f1" (UID: "123708b7-bba7-4cc1-a2c0-9f3887a742f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:36.597072 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:36.597043    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123708b7-bba7-4cc1-a2c0-9f3887a742f1-workload-socket" pod "123708b7-bba7-4cc1-a2c0-9f3887a742f1" (UID: "123708b7-bba7-4cc1-a2c0-9f3887a742f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:36.597235 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:36.597218    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123708b7-bba7-4cc1-a2c0-9f3887a742f1-istio-data" pod "123708b7-bba7-4cc1-a2c0-9f3887a742f1" (UID: "123708b7-bba7-4cc1-a2c0-9f3887a742f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:36.597390 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:36.597374    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123708b7-bba7-4cc1-a2c0-9f3887a742f1-workload-certs" pod "123708b7-bba7-4cc1-a2c0-9f3887a742f1" (UID: "123708b7-bba7-4cc1-a2c0-9f3887a742f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:36.598004 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:36.597974    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/123708b7-bba7-4cc1-a2c0-9f3887a742f1-istiod-ca-cert" pod "123708b7-bba7-4cc1-a2c0-9f3887a742f1" (UID: "123708b7-bba7-4cc1-a2c0-9f3887a742f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:17:36.601136 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:36.600075    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/123708b7-bba7-4cc1-a2c0-9f3887a742f1-kube-api-access-cmf82" pod "123708b7-bba7-4cc1-a2c0-9f3887a742f1" (UID: "123708b7-bba7-4cc1-a2c0-9f3887a742f1"). InnerVolumeSpecName "kube-api-access-cmf82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:36.603273 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:36.603240    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/123708b7-bba7-4cc1-a2c0-9f3887a742f1-istio-podinfo" pod "123708b7-bba7-4cc1-a2c0-9f3887a742f1" (UID: "123708b7-bba7-4cc1-a2c0-9f3887a742f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:17:36.604497 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:36.604466    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/123708b7-bba7-4cc1-a2c0-9f3887a742f1-istio-token" pod "123708b7-bba7-4cc1-a2c0-9f3887a742f1" (UID: "123708b7-bba7-4cc1-a2c0-9f3887a742f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:36.604992 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:36.604962    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123708b7-bba7-4cc1-a2c0-9f3887a742f1-istio-envoy" pod "123708b7-bba7-4cc1-a2c0-9f3887a742f1" (UID: "123708b7-bba7-4cc1-a2c0-9f3887a742f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:36.696951 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:36.696917    4393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/123708b7-bba7-4cc1-a2c0-9f3887a742f1-istio-podinfo\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:17:36.696951 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:36.696947    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmf82\" (UniqueName: \"kubernetes.io/projected/123708b7-bba7-4cc1-a2c0-9f3887a742f1-kube-api-access-cmf82\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:17:36.696951 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:36.696958    4393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/123708b7-bba7-4cc1-a2c0-9f3887a742f1-workload-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:17:36.697239 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:36.696967    4393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/123708b7-bba7-4cc1-a2c0-9f3887a742f1-istio-data\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:17:36.697239 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:36.696975    4393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/123708b7-bba7-4cc1-a2c0-9f3887a742f1-workload-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:17:36.697239 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:36.696983    4393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/123708b7-bba7-4cc1-a2c0-9f3887a742f1-istio-envoy\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:17:36.697239 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:36.696991    4393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/123708b7-bba7-4cc1-a2c0-9f3887a742f1-credential-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:17:36.697239 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:36.697000    4393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/123708b7-bba7-4cc1-a2c0-9f3887a742f1-istiod-ca-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:17:36.697239 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:36.697009    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/123708b7-bba7-4cc1-a2c0-9f3887a742f1-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:17:37.076248 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:37.076202    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc" event={"ID":"123708b7-bba7-4cc1-a2c0-9f3887a742f1","Type":"ContainerDied","Data":"c60ce85eb2210cb6634f9d1ba5fe4dc1b5572ffd6f78559e7ecf4f2e4827fe81"}
Oct 30 01:17:37.076764 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:37.076376    4393 scope.go:122] "RemoveContainer" containerID="2f2fdbbc17929bf3073d94f41ca2218364bf82daa842cdaf50abe3c6e0b1b13d"
Oct 30 01:17:37.077106 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:37.077082    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"
Oct 30 01:17:37.131479 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:37.131445    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"]
Oct 30 01:17:37.142853 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:37.142823    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce513-openshift-default-84cc9879-5drrc"]
Oct 30 01:17:37.753605 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:37.753566    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="123708b7-bba7-4cc1-a2c0-9f3887a742f1" path="/var/lib/kubelet/pods/123708b7-bba7-4cc1-a2c0-9f3887a742f1/volumes"
Oct 30 01:17:38.035931 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:38.035861    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf"
Oct 30 01:17:39.049803 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.046149    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 01:17:39.096981 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.096945    4393 generic.go:356] "Generic (PLEG): container finished" podID="d4370a67-835f-4d52-97c4-0419593f7975" containerID="8691ab65bf40b8deec2745258bb8493da0eccda6f7bb51ffab2ce426d9a409e9" exitCode=0
Oct 30 01:17:39.097164 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.097009    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"
Oct 30 01:17:39.097164 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.097024    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x" event={"ID":"d4370a67-835f-4d52-97c4-0419593f7975","Type":"ContainerDied","Data":"8691ab65bf40b8deec2745258bb8493da0eccda6f7bb51ffab2ce426d9a409e9"}
Oct 30 01:17:39.097164 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.097078    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x" event={"ID":"d4370a67-835f-4d52-97c4-0419593f7975","Type":"ContainerDied","Data":"aab546392622c12ca1dbda890ab241f7fee598daf9efedcec8d638ffb5b62ba4"}
Oct 30 01:17:39.097164 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.097107    4393 scope.go:122] "RemoveContainer" containerID="8691ab65bf40b8deec2745258bb8493da0eccda6f7bb51ffab2ce426d9a409e9"
Oct 30 01:17:39.111142 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.111116    4393 scope.go:122] "RemoveContainer" containerID="7c0f72c0435c5339acec126106d9a6d5cd3b794822b7605714567177c74a3621"
Oct 30 01:17:39.133953 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.133924    4393 scope.go:122] "RemoveContainer" containerID="8691ab65bf40b8deec2745258bb8493da0eccda6f7bb51ffab2ce426d9a409e9"
Oct 30 01:17:39.134938 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:17:39.134905    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8691ab65bf40b8deec2745258bb8493da0eccda6f7bb51ffab2ce426d9a409e9\": container with ID starting with 8691ab65bf40b8deec2745258bb8493da0eccda6f7bb51ffab2ce426d9a409e9 not found: ID does not exist" containerID="8691ab65bf40b8deec2745258bb8493da0eccda6f7bb51ffab2ce426d9a409e9"
Oct 30 01:17:39.135041 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.134945    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8691ab65bf40b8deec2745258bb8493da0eccda6f7bb51ffab2ce426d9a409e9"} err="failed to get container status \"8691ab65bf40b8deec2745258bb8493da0eccda6f7bb51ffab2ce426d9a409e9\": rpc error: code = NotFound desc = could not find container \"8691ab65bf40b8deec2745258bb8493da0eccda6f7bb51ffab2ce426d9a409e9\": container with ID starting with 8691ab65bf40b8deec2745258bb8493da0eccda6f7bb51ffab2ce426d9a409e9 not found: ID does not exist"
Oct 30 01:17:39.135041 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.134967    4393 scope.go:122] "RemoveContainer" containerID="7c0f72c0435c5339acec126106d9a6d5cd3b794822b7605714567177c74a3621"
Oct 30 01:17:39.135528 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:17:39.135506    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c0f72c0435c5339acec126106d9a6d5cd3b794822b7605714567177c74a3621\": container with ID starting with 7c0f72c0435c5339acec126106d9a6d5cd3b794822b7605714567177c74a3621 not found: ID does not exist" containerID="7c0f72c0435c5339acec126106d9a6d5cd3b794822b7605714567177c74a3621"
Oct 30 01:17:39.135644 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.135532    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c0f72c0435c5339acec126106d9a6d5cd3b794822b7605714567177c74a3621"} err="failed to get container status \"7c0f72c0435c5339acec126106d9a6d5cd3b794822b7605714567177c74a3621\": rpc error: code = NotFound desc = could not find container \"7c0f72c0435c5339acec126106d9a6d5cd3b794822b7605714567177c74a3621\": container with ID starting with 7c0f72c0435c5339acec126106d9a6d5cd3b794822b7605714567177c74a3621 not found: ID does not exist"
Oct 30 01:17:39.169876 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.169770    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4370a67-835f-4d52-97c4-0419593f7975-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4370a67-835f-4d52-97c4-0419593f7975-istiod-ca-cert\") pod \"d4370a67-835f-4d52-97c4-0419593f7975\" (UID: \"d4370a67-835f-4d52-97c4-0419593f7975\") "
Oct 30 01:17:39.170160 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.170136    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4370a67-835f-4d52-97c4-0419593f7975-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4370a67-835f-4d52-97c4-0419593f7975-workload-certs\") pod \"d4370a67-835f-4d52-97c4-0419593f7975\" (UID: \"d4370a67-835f-4d52-97c4-0419593f7975\") "
Oct 30 01:17:39.170686 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.170661    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4370a67-835f-4d52-97c4-0419593f7975-istio-token\" (UniqueName: \"kubernetes.io/projected/d4370a67-835f-4d52-97c4-0419593f7975-istio-token\") pod \"d4370a67-835f-4d52-97c4-0419593f7975\" (UID: \"d4370a67-835f-4d52-97c4-0419593f7975\") "
Oct 30 01:17:39.170889 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.170268    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4370a67-835f-4d52-97c4-0419593f7975-istiod-ca-cert" pod "d4370a67-835f-4d52-97c4-0419593f7975" (UID: "d4370a67-835f-4d52-97c4-0419593f7975"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:17:39.170987 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.170562    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4370a67-835f-4d52-97c4-0419593f7975-workload-certs" pod "d4370a67-835f-4d52-97c4-0419593f7975" (UID: "d4370a67-835f-4d52-97c4-0419593f7975"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:39.171078 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.171058    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4370a67-835f-4d52-97c4-0419593f7975-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4370a67-835f-4d52-97c4-0419593f7975-istio-podinfo\") pod \"d4370a67-835f-4d52-97c4-0419593f7975\" (UID: \"d4370a67-835f-4d52-97c4-0419593f7975\") "
Oct 30 01:17:39.171264 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.171247    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4370a67-835f-4d52-97c4-0419593f7975-kube-api-access-l2w6b\" (UniqueName: \"kubernetes.io/projected/d4370a67-835f-4d52-97c4-0419593f7975-kube-api-access-l2w6b\") pod \"d4370a67-835f-4d52-97c4-0419593f7975\" (UID: \"d4370a67-835f-4d52-97c4-0419593f7975\") "
Oct 30 01:17:39.171447 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.171429    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4370a67-835f-4d52-97c4-0419593f7975-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4370a67-835f-4d52-97c4-0419593f7975-istio-data\") pod \"d4370a67-835f-4d52-97c4-0419593f7975\" (UID: \"d4370a67-835f-4d52-97c4-0419593f7975\") "
Oct 30 01:17:39.171606 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.171591    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4370a67-835f-4d52-97c4-0419593f7975-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4370a67-835f-4d52-97c4-0419593f7975-workload-socket\") pod \"d4370a67-835f-4d52-97c4-0419593f7975\" (UID: \"d4370a67-835f-4d52-97c4-0419593f7975\") "
Oct 30 01:17:39.171751 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.171734    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4370a67-835f-4d52-97c4-0419593f7975-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4370a67-835f-4d52-97c4-0419593f7975-credential-socket\") pod \"d4370a67-835f-4d52-97c4-0419593f7975\" (UID: \"d4370a67-835f-4d52-97c4-0419593f7975\") "
Oct 30 01:17:39.171976 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.171958    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4370a67-835f-4d52-97c4-0419593f7975-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4370a67-835f-4d52-97c4-0419593f7975-istio-envoy\") pod \"d4370a67-835f-4d52-97c4-0419593f7975\" (UID: \"d4370a67-835f-4d52-97c4-0419593f7975\") "
Oct 30 01:17:39.172268 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.171896    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4370a67-835f-4d52-97c4-0419593f7975-istio-data" pod "d4370a67-835f-4d52-97c4-0419593f7975" (UID: "d4370a67-835f-4d52-97c4-0419593f7975"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:39.172353 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.172073    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4370a67-835f-4d52-97c4-0419593f7975-workload-socket" pod "d4370a67-835f-4d52-97c4-0419593f7975" (UID: "d4370a67-835f-4d52-97c4-0419593f7975"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:39.172353 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.172236    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4370a67-835f-4d52-97c4-0419593f7975-credential-socket" pod "d4370a67-835f-4d52-97c4-0419593f7975" (UID: "d4370a67-835f-4d52-97c4-0419593f7975"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:39.173324 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.173298    4393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4370a67-835f-4d52-97c4-0419593f7975-istio-data\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:17:39.173427 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.173328    4393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4370a67-835f-4d52-97c4-0419593f7975-workload-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:17:39.173427 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.173342    4393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4370a67-835f-4d52-97c4-0419593f7975-credential-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:17:39.173427 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.173356    4393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4370a67-835f-4d52-97c4-0419593f7975-istiod-ca-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:17:39.173427 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.173369    4393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4370a67-835f-4d52-97c4-0419593f7975-workload-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:17:39.176264 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.176214    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4370a67-835f-4d52-97c4-0419593f7975-istio-podinfo" pod "d4370a67-835f-4d52-97c4-0419593f7975" (UID: "d4370a67-835f-4d52-97c4-0419593f7975"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:17:39.176264 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.176223    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4370a67-835f-4d52-97c4-0419593f7975-istio-envoy" pod "d4370a67-835f-4d52-97c4-0419593f7975" (UID: "d4370a67-835f-4d52-97c4-0419593f7975"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:39.177966 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.177940    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4370a67-835f-4d52-97c4-0419593f7975-istio-token" pod "d4370a67-835f-4d52-97c4-0419593f7975" (UID: "d4370a67-835f-4d52-97c4-0419593f7975"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:39.178112 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.178092    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4370a67-835f-4d52-97c4-0419593f7975-kube-api-access-l2w6b" pod "d4370a67-835f-4d52-97c4-0419593f7975" (UID: "d4370a67-835f-4d52-97c4-0419593f7975"). InnerVolumeSpecName "kube-api-access-l2w6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:39.274880 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.274851    4393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4370a67-835f-4d52-97c4-0419593f7975-istio-envoy\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:17:39.274880 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.274880    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4370a67-835f-4d52-97c4-0419593f7975-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:17:39.274880 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.274890    4393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4370a67-835f-4d52-97c4-0419593f7975-istio-podinfo\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:17:39.274880 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.274898    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2w6b\" (UniqueName: \"kubernetes.io/projected/d4370a67-835f-4d52-97c4-0419593f7975-kube-api-access-l2w6b\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:17:39.445965 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.445887    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"]
Oct 30 01:17:39.449893 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.449869    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-52650f-openshift-default-6f5fdf664f-8gg8x"]
Oct 30 01:17:39.726320 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:39.726236    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4370a67-835f-4d52-97c4-0419593f7975" path="/var/lib/kubelet/pods/d4370a67-835f-4d52-97c4-0419593f7975/volumes"
Oct 30 01:17:42.042656 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:42.041349    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-29be73-58c6d59956-2sszw"]
Oct 30 01:17:42.075842 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:42.075805    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-29be73-64579fcddc-7cr2l"]
Oct 30 01:17:47.042097 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:47.042049    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-29be73-58c6d59956-2sszw" podUID="7bd51483-6bfc-4535-bd66-4c4e74ca70c8" containerName="payload-processing" containerID="cri-o://ab90a4a48d48ca894dc293ecd89160bd83c83cdfb92c5de4e7b845a90520eba0" gracePeriod=25
Oct 30 01:17:47.076858 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:47.076820    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29be73-64579fcddc-7cr2l" podUID="993c1f81-d8b2-46ac-8a86-f6399168de03" containerName="payload-pre-processing" containerID="cri-o://9c84d4f6e1224738e6ca6042606ee0305cb0b739ce43476fa35e74af67074918" gracePeriod=25
Oct 30 01:17:47.234335 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:47.234297    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-29be73-58c6d59956-2sszw" podUID="7bd51483-6bfc-4535-bd66-4c4e74ca70c8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:17:51.292348 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:51.292304    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj" podUID="2fc7af8a-93f1-4792-b170-90c675c8a919" containerName="main" containerID="cri-o://3a20f1761ac7c5c9e24f6cc42fa65d4f5042d40568b3ebdc47e6df7f5bfd0bc9" gracePeriod=45
Oct 30 01:17:51.594422 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:51.594389    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj"
Oct 30 01:17:51.605316 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:51.605290    4393 scope.go:122] "RemoveContainer" containerID="178ead6272f08b75482d71381d3b6d6dfa2811d75d8c530769b6208542f13749"
Oct 30 01:17:51.617658 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:51.617523    4393 scope.go:122] "RemoveContainer" containerID="3a20f1761ac7c5c9e24f6cc42fa65d4f5042d40568b3ebdc47e6df7f5bfd0bc9"
Oct 30 01:17:51.726223 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:51.726189    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fc7af8a-93f1-4792-b170-90c675c8a919-tls-certs\" (UniqueName: \"kubernetes.io/secret/2fc7af8a-93f1-4792-b170-90c675c8a919-tls-certs\") pod \"2fc7af8a-93f1-4792-b170-90c675c8a919\" (UID: \"2fc7af8a-93f1-4792-b170-90c675c8a919\") "
Oct 30 01:17:51.726456 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:51.726438    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fc7af8a-93f1-4792-b170-90c675c8a919-kube-api-access-r6pp2\" (UniqueName: \"kubernetes.io/projected/2fc7af8a-93f1-4792-b170-90c675c8a919-kube-api-access-r6pp2\") pod \"2fc7af8a-93f1-4792-b170-90c675c8a919\" (UID: \"2fc7af8a-93f1-4792-b170-90c675c8a919\") "
Oct 30 01:17:51.726503 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:51.726489    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc7af8a-93f1-4792-b170-90c675c8a919-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fc7af8a-93f1-4792-b170-90c675c8a919-dshm\") pod \"2fc7af8a-93f1-4792-b170-90c675c8a919\" (UID: \"2fc7af8a-93f1-4792-b170-90c675c8a919\") "
Oct 30 01:17:51.726581 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:51.726557    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc7af8a-93f1-4792-b170-90c675c8a919-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fc7af8a-93f1-4792-b170-90c675c8a919-model-cache\") pod \"2fc7af8a-93f1-4792-b170-90c675c8a919\" (UID: \"2fc7af8a-93f1-4792-b170-90c675c8a919\") "
Oct 30 01:17:51.726640 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:51.726622    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc7af8a-93f1-4792-b170-90c675c8a919-home\" (UniqueName: \"kubernetes.io/empty-dir/2fc7af8a-93f1-4792-b170-90c675c8a919-home\") pod \"2fc7af8a-93f1-4792-b170-90c675c8a919\" (UID: \"2fc7af8a-93f1-4792-b170-90c675c8a919\") "
Oct 30 01:17:51.726706 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:51.726656    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fc7af8a-93f1-4792-b170-90c675c8a919-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fc7af8a-93f1-4792-b170-90c675c8a919-tmp-dir\") pod \"2fc7af8a-93f1-4792-b170-90c675c8a919\" (UID: \"2fc7af8a-93f1-4792-b170-90c675c8a919\") "
Oct 30 01:17:51.726956 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:51.726926    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc7af8a-93f1-4792-b170-90c675c8a919-model-cache" pod "2fc7af8a-93f1-4792-b170-90c675c8a919" (UID: "2fc7af8a-93f1-4792-b170-90c675c8a919"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:51.727247 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:51.727126    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc7af8a-93f1-4792-b170-90c675c8a919-home" pod "2fc7af8a-93f1-4792-b170-90c675c8a919" (UID: "2fc7af8a-93f1-4792-b170-90c675c8a919"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:51.727247 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:51.727135    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc7af8a-93f1-4792-b170-90c675c8a919-tmp-dir" pod "2fc7af8a-93f1-4792-b170-90c675c8a919" (UID: "2fc7af8a-93f1-4792-b170-90c675c8a919"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:51.727435 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:51.727309    4393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fc7af8a-93f1-4792-b170-90c675c8a919-model-cache\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:17:51.727435 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:51.727328    4393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fc7af8a-93f1-4792-b170-90c675c8a919-home\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:17:51.727435 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:51.727357    4393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fc7af8a-93f1-4792-b170-90c675c8a919-tmp-dir\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:17:51.729370 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:51.729343    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fc7af8a-93f1-4792-b170-90c675c8a919-kube-api-access-r6pp2" pod "2fc7af8a-93f1-4792-b170-90c675c8a919" (UID: "2fc7af8a-93f1-4792-b170-90c675c8a919"). InnerVolumeSpecName "kube-api-access-r6pp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:17:51.729556 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:51.729531    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fc7af8a-93f1-4792-b170-90c675c8a919-dshm" pod "2fc7af8a-93f1-4792-b170-90c675c8a919" (UID: "2fc7af8a-93f1-4792-b170-90c675c8a919"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:17:51.729663 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:51.729577    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fc7af8a-93f1-4792-b170-90c675c8a919-tls-certs" pod "2fc7af8a-93f1-4792-b170-90c675c8a919" (UID: "2fc7af8a-93f1-4792-b170-90c675c8a919"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:17:51.829295 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:51.829194    4393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fc7af8a-93f1-4792-b170-90c675c8a919-tls-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:17:51.829295 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:51.829239    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6pp2\" (UniqueName: \"kubernetes.io/projected/2fc7af8a-93f1-4792-b170-90c675c8a919-kube-api-access-r6pp2\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:17:51.829295 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:51.829257    4393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fc7af8a-93f1-4792-b170-90c675c8a919-dshm\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:17:52.218000 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:52.217960    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj" event={"ID":"2fc7af8a-93f1-4792-b170-90c675c8a919","Type":"ContainerDied","Data":"3a20f1761ac7c5c9e24f6cc42fa65d4f5042d40568b3ebdc47e6df7f5bfd0bc9"}
Oct 30 01:17:52.218000 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:52.217992    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj"
Oct 30 01:17:52.218250 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:52.218015    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj" event={"ID":"2fc7af8a-93f1-4792-b170-90c675c8a919","Type":"ContainerDied","Data":"89b9277ce05c03cc39beb0541ae61c491e5f60502d705fd82e00520785164e82"}
Oct 30 01:17:52.295768 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:52.295731    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj"]
Oct 30 01:17:52.299928 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:52.299904    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-29be73/distinct-w1-29be73-kserve-867cdb7458-ncqxj"]
Oct 30 01:17:53.728054 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:53.728022    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fc7af8a-93f1-4792-b170-90c675c8a919" path="/var/lib/kubelet/pods/2fc7af8a-93f1-4792-b170-90c675c8a919/volumes"
Oct 30 01:17:56.218815 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:56.218757    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29be73-64579fcddc-7cr2l" podUID="993c1f81-d8b2-46ac-8a86-f6399168de03" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:17:56.512488 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:17:56.512398    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:17:57.256079 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:17:57.256031    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-29be73-58c6d59956-2sszw" podUID="7bd51483-6bfc-4535-bd66-4c4e74ca70c8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:18:06.219493 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:06.219450    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29be73-64579fcddc-7cr2l" podUID="993c1f81-d8b2-46ac-8a86-f6399168de03" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:18:07.234147 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:18:07.234116    4393 logging.go:55] [core] [Channel #9764 SubChannel #9765]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.63:50052", ServerName: "10.133.0.63:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.63:50052: connect: connection refused"
Oct 30 01:18:07.373273 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:07.373236    4393 generic.go:356] "Generic (PLEG): container finished" podID="993c1f81-d8b2-46ac-8a86-f6399168de03" containerID="9c84d4f6e1224738e6ca6042606ee0305cb0b739ce43476fa35e74af67074918" exitCode=0
Oct 30 01:18:07.373444 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:07.373361    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29be73-64579fcddc-7cr2l" event={"ID":"993c1f81-d8b2-46ac-8a86-f6399168de03","Type":"ContainerDied","Data":"9c84d4f6e1224738e6ca6042606ee0305cb0b739ce43476fa35e74af67074918"}
Oct 30 01:18:07.374991 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:07.374961    4393 generic.go:356] "Generic (PLEG): container finished" podID="7bd51483-6bfc-4535-bd66-4c4e74ca70c8" containerID="ab90a4a48d48ca894dc293ecd89160bd83c83cdfb92c5de4e7b845a90520eba0" exitCode=0
Oct 30 01:18:07.375133 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:07.375031    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-29be73-58c6d59956-2sszw" event={"ID":"7bd51483-6bfc-4535-bd66-4c4e74ca70c8","Type":"ContainerDied","Data":"ab90a4a48d48ca894dc293ecd89160bd83c83cdfb92c5de4e7b845a90520eba0"}
Oct 30 01:18:07.375133 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:07.375080    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-29be73-58c6d59956-2sszw" event={"ID":"7bd51483-6bfc-4535-bd66-4c4e74ca70c8","Type":"ContainerDied","Data":"57abb6ac472a9e6ff33eaebc317c13717b9a8ff9b93e3ea45651be2e13b9c7be"}
Oct 30 01:18:07.375133 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:07.375096    4393 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57abb6ac472a9e6ff33eaebc317c13717b9a8ff9b93e3ea45651be2e13b9c7be"
Oct 30 01:18:07.388189 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:07.388160    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-29be73-58c6d59956-2sszw"
Oct 30 01:18:07.427290 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:07.427268    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29be73-64579fcddc-7cr2l"
Oct 30 01:18:07.485011 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:07.484982    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bd51483-6bfc-4535-bd66-4c4e74ca70c8-kube-api-access-zczhh\" (UniqueName: \"kubernetes.io/projected/7bd51483-6bfc-4535-bd66-4c4e74ca70c8-kube-api-access-zczhh\") pod \"7bd51483-6bfc-4535-bd66-4c4e74ca70c8\" (UID: \"7bd51483-6bfc-4535-bd66-4c4e74ca70c8\") "
Oct 30 01:18:07.485254 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:07.485113    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/993c1f81-d8b2-46ac-8a86-f6399168de03-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/993c1f81-d8b2-46ac-8a86-f6399168de03-plugins-config-volume\") pod \"993c1f81-d8b2-46ac-8a86-f6399168de03\" (UID: \"993c1f81-d8b2-46ac-8a86-f6399168de03\") "
Oct 30 01:18:07.485254 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:07.485184    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bd51483-6bfc-4535-bd66-4c4e74ca70c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd51483-6bfc-4535-bd66-4c4e74ca70c8-plugins-config-volume\") pod \"7bd51483-6bfc-4535-bd66-4c4e74ca70c8\" (UID: \"7bd51483-6bfc-4535-bd66-4c4e74ca70c8\") "
Oct 30 01:18:07.485399 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:07.485256    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/993c1f81-d8b2-46ac-8a86-f6399168de03-kube-api-access-pc4tb\" (UniqueName: \"kubernetes.io/projected/993c1f81-d8b2-46ac-8a86-f6399168de03-kube-api-access-pc4tb\") pod \"993c1f81-d8b2-46ac-8a86-f6399168de03\" (UID: \"993c1f81-d8b2-46ac-8a86-f6399168de03\") "
Oct 30 01:18:07.485668 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:07.485608    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/993c1f81-d8b2-46ac-8a86-f6399168de03-plugins-config-volume" pod "993c1f81-d8b2-46ac-8a86-f6399168de03" (UID: "993c1f81-d8b2-46ac-8a86-f6399168de03"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:18:07.485795 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:07.485682    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd51483-6bfc-4535-bd66-4c4e74ca70c8-plugins-config-volume" pod "7bd51483-6bfc-4535-bd66-4c4e74ca70c8" (UID: "7bd51483-6bfc-4535-bd66-4c4e74ca70c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:18:07.486167 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:07.486144    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/993c1f81-d8b2-46ac-8a86-f6399168de03-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:18:07.486281 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:07.486171    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd51483-6bfc-4535-bd66-4c4e74ca70c8-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:18:07.487626 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:07.487600    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/993c1f81-d8b2-46ac-8a86-f6399168de03-kube-api-access-pc4tb" pod "993c1f81-d8b2-46ac-8a86-f6399168de03" (UID: "993c1f81-d8b2-46ac-8a86-f6399168de03"). InnerVolumeSpecName "kube-api-access-pc4tb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:07.487992 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:07.487966    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd51483-6bfc-4535-bd66-4c4e74ca70c8-kube-api-access-zczhh" pod "7bd51483-6bfc-4535-bd66-4c4e74ca70c8" (UID: "7bd51483-6bfc-4535-bd66-4c4e74ca70c8"). InnerVolumeSpecName "kube-api-access-zczhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:07.588071 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:07.587989    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc4tb\" (UniqueName: \"kubernetes.io/projected/993c1f81-d8b2-46ac-8a86-f6399168de03-kube-api-access-pc4tb\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:18:07.588071 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:07.588019    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zczhh\" (UniqueName: \"kubernetes.io/projected/7bd51483-6bfc-4535-bd66-4c4e74ca70c8-kube-api-access-zczhh\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:18:08.233879 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:08.233836    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-29be73-58c6d59956-2sszw" podUID="7bd51483-6bfc-4535-bd66-4c4e74ca70c8" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.63:50052\" within 1s: context deadline exceeded"
Oct 30 01:18:08.234075 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:08.233931    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-29be73-58c6d59956-2sszw"
Oct 30 01:18:08.394246 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:08.394214    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-29be73-58c6d59956-2sszw"
Oct 30 01:18:08.394246 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:08.394229    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29be73-64579fcddc-7cr2l" event={"ID":"993c1f81-d8b2-46ac-8a86-f6399168de03","Type":"ContainerDied","Data":"a3b161b83c062a92cd7fca9ea5af231dab115054705aa1e4e55be8af2fba2978"}
Oct 30 01:18:08.394888 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:08.394215    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-29be73-64579fcddc-7cr2l"
Oct 30 01:18:08.394888 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:08.394314    4393 scope.go:122] "RemoveContainer" containerID="9c84d4f6e1224738e6ca6042606ee0305cb0b739ce43476fa35e74af67074918"
Oct 30 01:18:08.459952 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:08.459920    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-29be73-58c6d59956-2sszw"]
Oct 30 01:18:08.471707 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:08.471682    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-29be73-58c6d59956-2sszw"]
Oct 30 01:18:08.498608 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:08.498568    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-29be73-64579fcddc-7cr2l"]
Oct 30 01:18:08.504813 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:08.504768    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-29be73-64579fcddc-7cr2l"]
Oct 30 01:18:09.726658 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:09.726621    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bd51483-6bfc-4535-bd66-4c4e74ca70c8" path="/var/lib/kubelet/pods/7bd51483-6bfc-4535-bd66-4c4e74ca70c8/volumes"
Oct 30 01:18:09.727243 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:09.727223    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="993c1f81-d8b2-46ac-8a86-f6399168de03" path="/var/lib/kubelet/pods/993c1f81-d8b2-46ac-8a86-f6399168de03/volumes"
Oct 30 01:18:16.975686 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:16.975646    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm"]
Oct 30 01:18:16.976469 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:16.976450    4393 cpu_manager.go:414] "containerMap: removing container" podUID="993c1f81-d8b2-46ac-8a86-f6399168de03" containerName="payload-pre-processing"
Oct 30 01:18:16.976515 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:16.976471    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="993c1f81-d8b2-46ac-8a86-f6399168de03" containerName="payload-pre-processing"
Oct 30 01:18:16.976515 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:16.976488    4393 cpu_manager.go:414] "containerMap: removing container" podUID="2fc7af8a-93f1-4792-b170-90c675c8a919" containerName="main"
Oct 30 01:18:16.976515 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:16.976493    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fc7af8a-93f1-4792-b170-90c675c8a919" containerName="main"
Oct 30 01:18:16.976615 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:16.976524    4393 cpu_manager.go:414] "containerMap: removing container" podUID="123708b7-bba7-4cc1-a2c0-9f3887a742f1" containerName="istio-proxy"
Oct 30 01:18:16.976615 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:16.976529    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="123708b7-bba7-4cc1-a2c0-9f3887a742f1" containerName="istio-proxy"
Oct 30 01:18:16.976615 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:16.976544    4393 cpu_manager.go:414] "containerMap: removing container" podUID="d4370a67-835f-4d52-97c4-0419593f7975" containerName="istio-proxy"
Oct 30 01:18:16.976615 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:16.976548    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4370a67-835f-4d52-97c4-0419593f7975" containerName="istio-proxy"
Oct 30 01:18:16.976615 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:16.976558    4393 cpu_manager.go:414] "containerMap: removing container" podUID="d4370a67-835f-4d52-97c4-0419593f7975" containerName="istio-proxy"
Oct 30 01:18:16.976615 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:16.976564    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4370a67-835f-4d52-97c4-0419593f7975" containerName="istio-proxy"
Oct 30 01:18:16.976615 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:16.976574    4393 cpu_manager.go:414] "containerMap: removing container" podUID="7bd51483-6bfc-4535-bd66-4c4e74ca70c8" containerName="payload-processing"
Oct 30 01:18:16.976615 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:16.976579    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bd51483-6bfc-4535-bd66-4c4e74ca70c8" containerName="payload-processing"
Oct 30 01:18:16.976895 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:16.976676    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="123708b7-bba7-4cc1-a2c0-9f3887a742f1" containerName="istio-proxy"
Oct 30 01:18:16.976895 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:16.976684    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4370a67-835f-4d52-97c4-0419593f7975" containerName="istio-proxy"
Oct 30 01:18:16.976895 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:16.976692    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4370a67-835f-4d52-97c4-0419593f7975" containerName="istio-proxy"
Oct 30 01:18:16.976895 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:16.976699    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bd51483-6bfc-4535-bd66-4c4e74ca70c8" containerName="payload-processing"
Oct 30 01:18:16.976895 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:16.976706    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="993c1f81-d8b2-46ac-8a86-f6399168de03" containerName="payload-pre-processing"
Oct 30 01:18:16.976895 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:16.976713    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fc7af8a-93f1-4792-b170-90c675c8a919" containerName="main"
Oct 30 01:18:16.982069 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:16.982046    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm"
Oct 30 01:18:17.001152 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:17.001119    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm"]
Oct 30 01:18:17.006262 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:17.006234    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-b73232-serving-cert\""
Oct 30 01:18:17.136601 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:17.136570    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2526f9aa-9b27-4a09-b321-ec07b6b0131e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm\" (UID: \"2526f9aa-9b27-4a09-b321-ec07b6b0131e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm"
Oct 30 01:18:17.136848 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:17.136615    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2526f9aa-9b27-4a09-b321-ec07b6b0131e-metrics-certs\") pod \"maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm\" (UID: \"2526f9aa-9b27-4a09-b321-ec07b6b0131e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm"
Oct 30 01:18:17.136848 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:17.136815    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2526f9aa-9b27-4a09-b321-ec07b6b0131e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm\" (UID: \"2526f9aa-9b27-4a09-b321-ec07b6b0131e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm"
Oct 30 01:18:17.136848 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:17.136842    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl9kg\" (UniqueName: \"kubernetes.io/projected/2526f9aa-9b27-4a09-b321-ec07b6b0131e-kube-api-access-xl9kg\") pod \"maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm\" (UID: \"2526f9aa-9b27-4a09-b321-ec07b6b0131e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm"
Oct 30 01:18:17.238613 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:17.238491    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2526f9aa-9b27-4a09-b321-ec07b6b0131e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm\" (UID: \"2526f9aa-9b27-4a09-b321-ec07b6b0131e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm"
Oct 30 01:18:17.238613 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:17.238566    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl9kg\" (UniqueName: \"kubernetes.io/projected/2526f9aa-9b27-4a09-b321-ec07b6b0131e-kube-api-access-xl9kg\") pod \"maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm\" (UID: \"2526f9aa-9b27-4a09-b321-ec07b6b0131e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm"
Oct 30 01:18:17.239010 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:18:17.238683    4393 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73232-serving-cert: secret "maas-api-e2e-ipp-a-b73232-serving-cert" not found
Oct 30 01:18:17.239010 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:18:17.238765    4393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2526f9aa-9b27-4a09-b321-ec07b6b0131e-maas-api-tls podName:2526f9aa-9b27-4a09-b321-ec07b6b0131e nodeName:}" failed. No retries permitted until 2026-10-30 01:18:17.738742152 +0000 UTC m=+2788.126418539 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2526f9aa-9b27-4a09-b321-ec07b6b0131e-maas-api-tls") pod "maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm" (UID: "2526f9aa-9b27-4a09-b321-ec07b6b0131e") : secret "maas-api-e2e-ipp-a-b73232-serving-cert" not found
Oct 30 01:18:17.239010 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:17.238864    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2526f9aa-9b27-4a09-b321-ec07b6b0131e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm\" (UID: \"2526f9aa-9b27-4a09-b321-ec07b6b0131e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm"
Oct 30 01:18:17.239010 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:17.238904    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2526f9aa-9b27-4a09-b321-ec07b6b0131e-metrics-certs\") pod \"maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm\" (UID: \"2526f9aa-9b27-4a09-b321-ec07b6b0131e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm"
Oct 30 01:18:17.240130 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:17.240105    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2526f9aa-9b27-4a09-b321-ec07b6b0131e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm\" (UID: \"2526f9aa-9b27-4a09-b321-ec07b6b0131e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm"
Oct 30 01:18:17.242358 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:17.242333    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2526f9aa-9b27-4a09-b321-ec07b6b0131e-metrics-certs\") pod \"maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm\" (UID: \"2526f9aa-9b27-4a09-b321-ec07b6b0131e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm"
Oct 30 01:18:17.265599 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:17.265565    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl9kg\" (UniqueName: \"kubernetes.io/projected/2526f9aa-9b27-4a09-b321-ec07b6b0131e-kube-api-access-xl9kg\") pod \"maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm\" (UID: \"2526f9aa-9b27-4a09-b321-ec07b6b0131e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm"
Oct 30 01:18:17.746080 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:17.746047    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2526f9aa-9b27-4a09-b321-ec07b6b0131e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm\" (UID: \"2526f9aa-9b27-4a09-b321-ec07b6b0131e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm"
Oct 30 01:18:17.748945 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:17.748921    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2526f9aa-9b27-4a09-b321-ec07b6b0131e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm\" (UID: \"2526f9aa-9b27-4a09-b321-ec07b6b0131e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm"
Oct 30 01:18:17.898820 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:17.898773    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm"
Oct 30 01:18:18.204083 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:18.204054    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm"]
Oct 30 01:18:18.204449 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:18:18.204259    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2526f9aa_9b27_4a09_b321_ec07b6b0131e.slice/crio-b51b2381a5ba1ecb68f5cb373185236738618e86c5181f418b8e8390bf1546cf WatchSource:0}: Error finding container b51b2381a5ba1ecb68f5cb373185236738618e86c5181f418b8e8390bf1546cf: Status 404 returned error can't find the container with id b51b2381a5ba1ecb68f5cb373185236738618e86c5181f418b8e8390bf1546cf
Oct 30 01:18:18.473672 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:18.473579    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm" event={"ID":"2526f9aa-9b27-4a09-b321-ec07b6b0131e","Type":"ContainerStarted","Data":"b51b2381a5ba1ecb68f5cb373185236738618e86c5181f418b8e8390bf1546cf"}
Oct 30 01:18:19.482280 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:19.482242    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm" event={"ID":"2526f9aa-9b27-4a09-b321-ec07b6b0131e","Type":"ContainerStarted","Data":"9363b2c1be68aada576269291ab458aad0361a17baeb10a5494bd9faf6179fd7"}
Oct 30 01:18:19.482846 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:19.482341    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm"
Oct 30 01:18:19.526765 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:19.526711    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm" podStartSLOduration=3.204775988 podStartE2EDuration="3.526696679s" totalImagesPullingTime="321.920691ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:18:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:18:19.522159793 +0000 UTC m=+2789.909836182" watchObservedRunningTime="2026-10-30 01:18:19.526696679 +0000 UTC m=+2789.914373113"
Oct 30 01:18:25.497201 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:25.497169    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm"
Oct 30 01:18:26.513494 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:18:26.513457    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:18:27.569388 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:27.569359    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-l4gst_9217cef0-6bf2-41d5-b303-4c4b78dcb7f0/payload-processing/0.log"
Oct 30 01:18:27.767412 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:27.767378    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-l4gst_9217cef0-6bf2-41d5-b303-4c4b78dcb7f0/payload-processing/0.log"
Oct 30 01:18:38.407551 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:38.407515    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4"]
Oct 30 01:18:38.408365 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:38.407719    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4" podUID="44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e" containerName="maas-api" containerID="cri-o://5cf540c543d692e86ac2d45b39fdf22dcd7fd71ed7cae3ad9024f26399b588d2" gracePeriod=30
Oct 30 01:18:38.646414 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:38.646375    4393 generic.go:356] "Generic (PLEG): container finished" podID="44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e" containerID="5cf540c543d692e86ac2d45b39fdf22dcd7fd71ed7cae3ad9024f26399b588d2" exitCode=0
Oct 30 01:18:38.646602 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:38.646436    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4" event={"ID":"44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e","Type":"ContainerDied","Data":"5cf540c543d692e86ac2d45b39fdf22dcd7fd71ed7cae3ad9024f26399b588d2"}
Oct 30 01:18:38.756558 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:38.756531    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4"
Oct 30 01:18:38.817528 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:38.817489    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e-openshift-service-ca\") pod \"44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e\" (UID: \"44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e\") "
Oct 30 01:18:38.817806 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:38.817664    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e-metrics-certs\") pod \"44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e\" (UID: \"44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e\") "
Oct 30 01:18:38.817806 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:38.817704    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e-maas-api-tls\") pod \"44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e\" (UID: \"44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e\") "
Oct 30 01:18:38.817973 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:38.817835    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e-kube-api-access-76wwd\" (UniqueName: \"kubernetes.io/projected/44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e-kube-api-access-76wwd\") pod \"44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e\" (UID: \"44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e\") "
Oct 30 01:18:38.819229 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:38.819197    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e-openshift-service-ca" pod "44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e" (UID: "44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:18:38.825206 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:38.825163    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e-metrics-certs" pod "44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e" (UID: "44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:18:38.825206 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:38.825191    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e-kube-api-access-76wwd" pod "44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e" (UID: "44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e"). InnerVolumeSpecName "kube-api-access-76wwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:38.826847 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:38.826772    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e-maas-api-tls" pod "44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e" (UID: "44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:18:38.920048 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:38.919975    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76wwd\" (UniqueName: \"kubernetes.io/projected/44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e-kube-api-access-76wwd\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:18:38.920048 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:38.920008    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:18:38.920048 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:38.920022    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:18:38.920048 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:38.920036    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:18:39.656259 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:39.656219    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4" event={"ID":"44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e","Type":"ContainerDied","Data":"70cb321491fff771712378fca2869d978a169068ec7148f85da8bbc640e776e6"}
Oct 30 01:18:39.656259 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:39.656237    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4"
Oct 30 01:18:39.656805 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:39.656277    4393 scope.go:122] "RemoveContainer" containerID="5cf540c543d692e86ac2d45b39fdf22dcd7fd71ed7cae3ad9024f26399b588d2"
Oct 30 01:18:39.708403 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:39.708323    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4"]
Oct 30 01:18:39.734377 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:39.734342    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-29be73-6fb858b479-q8sx4"]
Oct 30 01:18:41.727013 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:41.726979    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e" path="/var/lib/kubelet/pods/44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e/volumes"
Oct 30 01:18:43.129860 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:43.125411    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-6f5b8674dc-xch8n"]
Oct 30 01:18:43.129860 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:43.126602    4393 cpu_manager.go:414] "containerMap: removing container" podUID="44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e" containerName="maas-api"
Oct 30 01:18:43.129860 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:43.126623    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e" containerName="maas-api"
Oct 30 01:18:43.147806 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:43.132931    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="44b4eed1-220e-4ff5-a0c2-90a1ccc1d92e" containerName="maas-api"
Oct 30 01:18:43.147806 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:43.146640    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-6f5b8674dc-xch8n"
Oct 30 01:18:43.153941 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:43.153907    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-6f5b8674dc-xch8n"]
Oct 30 01:18:43.194134 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:43.192040    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9a8001\""
Oct 30 01:18:43.195140 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:43.195111    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-9a8001-dockercfg-9t4zv\""
Oct 30 01:18:43.296453 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:43.296422    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9a8001-646fd774b6-5xz5c"]
Oct 30 01:18:43.299572 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:43.299544    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q628\" (UniqueName: \"kubernetes.io/projected/5b80809c-0baa-4d5c-81c5-ea61440aadff-kube-api-access-9q628\") pod \"payload-pre-processing-e2e-ipp-b-9a8001-6f5b8674dc-xch8n\" (UID: \"5b80809c-0baa-4d5c-81c5-ea61440aadff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-6f5b8674dc-xch8n"
Oct 30 01:18:43.299735 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:43.299688    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b80809c-0baa-4d5c-81c5-ea61440aadff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9a8001-6f5b8674dc-xch8n\" (UID: \"5b80809c-0baa-4d5c-81c5-ea61440aadff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-6f5b8674dc-xch8n"
Oct 30 01:18:43.301290 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:43.301276    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9a8001-646fd774b6-5xz5c"
Oct 30 01:18:43.321418 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:43.321387    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9a8001-646fd774b6-5xz5c"]
Oct 30 01:18:43.401655 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:43.401574    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q628\" (UniqueName: \"kubernetes.io/projected/5b80809c-0baa-4d5c-81c5-ea61440aadff-kube-api-access-9q628\") pod \"payload-pre-processing-e2e-ipp-b-9a8001-6f5b8674dc-xch8n\" (UID: \"5b80809c-0baa-4d5c-81c5-ea61440aadff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-6f5b8674dc-xch8n"
Oct 30 01:18:43.401837 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:43.401687    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b80809c-0baa-4d5c-81c5-ea61440aadff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9a8001-6f5b8674dc-xch8n\" (UID: \"5b80809c-0baa-4d5c-81c5-ea61440aadff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-6f5b8674dc-xch8n"
Oct 30 01:18:43.401887 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:43.401870    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v7lf\" (UniqueName: \"kubernetes.io/projected/5428a426-9be0-41b6-b518-beaa791a21c7-kube-api-access-9v7lf\") pod \"payload-processing-e2e-ipp-b-9a8001-646fd774b6-5xz5c\" (UID: \"5428a426-9be0-41b6-b518-beaa791a21c7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9a8001-646fd774b6-5xz5c"
Oct 30 01:18:43.401951 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:43.401935    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5428a426-9be0-41b6-b518-beaa791a21c7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9a8001-646fd774b6-5xz5c\" (UID: \"5428a426-9be0-41b6-b518-beaa791a21c7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9a8001-646fd774b6-5xz5c"
Oct 30 01:18:43.402456 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:43.402435    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b80809c-0baa-4d5c-81c5-ea61440aadff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9a8001-6f5b8674dc-xch8n\" (UID: \"5b80809c-0baa-4d5c-81c5-ea61440aadff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-6f5b8674dc-xch8n"
Oct 30 01:18:43.424603 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:43.424572    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q628\" (UniqueName: \"kubernetes.io/projected/5b80809c-0baa-4d5c-81c5-ea61440aadff-kube-api-access-9q628\") pod \"payload-pre-processing-e2e-ipp-b-9a8001-6f5b8674dc-xch8n\" (UID: \"5b80809c-0baa-4d5c-81c5-ea61440aadff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-6f5b8674dc-xch8n"
Oct 30 01:18:43.503660 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:43.503609    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v7lf\" (UniqueName: \"kubernetes.io/projected/5428a426-9be0-41b6-b518-beaa791a21c7-kube-api-access-9v7lf\") pod \"payload-processing-e2e-ipp-b-9a8001-646fd774b6-5xz5c\" (UID: \"5428a426-9be0-41b6-b518-beaa791a21c7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9a8001-646fd774b6-5xz5c"
Oct 30 01:18:43.503852 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:43.503684    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5428a426-9be0-41b6-b518-beaa791a21c7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9a8001-646fd774b6-5xz5c\" (UID: \"5428a426-9be0-41b6-b518-beaa791a21c7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9a8001-646fd774b6-5xz5c"
Oct 30 01:18:43.504384 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:43.504362    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5428a426-9be0-41b6-b518-beaa791a21c7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9a8001-646fd774b6-5xz5c\" (UID: \"5428a426-9be0-41b6-b518-beaa791a21c7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9a8001-646fd774b6-5xz5c"
Oct 30 01:18:43.514152 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:43.514122    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-6f5b8674dc-xch8n"
Oct 30 01:18:43.526736 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:43.526716    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v7lf\" (UniqueName: \"kubernetes.io/projected/5428a426-9be0-41b6-b518-beaa791a21c7-kube-api-access-9v7lf\") pod \"payload-processing-e2e-ipp-b-9a8001-646fd774b6-5xz5c\" (UID: \"5428a426-9be0-41b6-b518-beaa791a21c7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9a8001-646fd774b6-5xz5c"
Oct 30 01:18:43.619639 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:43.619597    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9a8001-646fd774b6-5xz5c"
Oct 30 01:18:43.823657 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:43.823630    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-6f5b8674dc-xch8n"]
Oct 30 01:18:43.824062 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:18:43.824041    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b80809c_0baa_4d5c_81c5_ea61440aadff.slice/crio-b996858df314b07b68301964ef4f59b3dda82662cefcf1cb0227eba8250bf7d2 WatchSource:0}: Error finding container b996858df314b07b68301964ef4f59b3dda82662cefcf1cb0227eba8250bf7d2: Status 404 returned error can't find the container with id b996858df314b07b68301964ef4f59b3dda82662cefcf1cb0227eba8250bf7d2
Oct 30 01:18:43.970026 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:43.969068    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9a8001-646fd774b6-5xz5c"]
Oct 30 01:18:44.705944 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:44.705908    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-6f5b8674dc-xch8n" event={"ID":"5b80809c-0baa-4d5c-81c5-ea61440aadff","Type":"ContainerStarted","Data":"dd01657c7300c76c844b71398d2352fa5f6baa2d9844fdc5584a3371cce9f822"}
Oct 30 01:18:44.711566 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:44.705974    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-6f5b8674dc-xch8n" event={"ID":"5b80809c-0baa-4d5c-81c5-ea61440aadff","Type":"ContainerStarted","Data":"b996858df314b07b68301964ef4f59b3dda82662cefcf1cb0227eba8250bf7d2"}
Oct 30 01:18:44.711566 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:44.706065    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-6f5b8674dc-xch8n"
Oct 30 01:18:44.711566 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:44.708060    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9a8001-646fd774b6-5xz5c" event={"ID":"5428a426-9be0-41b6-b518-beaa791a21c7","Type":"ContainerStarted","Data":"2b6799735d132e44379e505e4384369d1592a910410879ec0eede729925ef1b5"}
Oct 30 01:18:44.711566 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:44.708106    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9a8001-646fd774b6-5xz5c" event={"ID":"5428a426-9be0-41b6-b518-beaa791a21c7","Type":"ContainerStarted","Data":"8a2ae59bc73452d245fdf99e72da19bce15db08b0f91011160e0b336ccb2f8ea"}
Oct 30 01:18:44.711566 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:44.708135    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9a8001-646fd774b6-5xz5c"
Oct 30 01:18:44.746825 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:44.746761    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-6f5b8674dc-xch8n" podStartSLOduration=1.74674305 podStartE2EDuration="1.74674305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:18:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:18:44.744458834 +0000 UTC m=+2815.132135222" watchObservedRunningTime="2026-10-30 01:18:44.74674305 +0000 UTC m=+2815.134419442"
Oct 30 01:18:44.786966 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:44.786915    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-9a8001-646fd774b6-5xz5c" podStartSLOduration=1.786901972 podStartE2EDuration="1.786901972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:18:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:18:44.783137018 +0000 UTC m=+2815.170813408" watchObservedRunningTime="2026-10-30 01:18:44.786901972 +0000 UTC m=+2815.174578396"
Oct 30 01:18:46.242356 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:46.242305    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"]
Oct 30 01:18:46.243055 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:46.242556    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx" podUID="914501a6-32b3-4158-9171-8320ce9291d4" containerName="istio-proxy" containerID="cri-o://4b3389fa5dc65f2f7f7e14ea8d839ab90e80abfba04a50a11cb7b2d0b8c01bc0" gracePeriod=30
Oct 30 01:18:46.728351 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:46.728309    4393 patch_prober.go:29] interesting pod/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 01:18:46.728543 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:46.728379    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx" podUID="914501a6-32b3-4158-9171-8320ce9291d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 01:18:51.530058 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:51.530033    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"
Oct 30 01:18:51.635611 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:51.635579    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914501a6-32b3-4158-9171-8320ce9291d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914501a6-32b3-4158-9171-8320ce9291d4-workload-socket\") pod \"914501a6-32b3-4158-9171-8320ce9291d4\" (UID: \"914501a6-32b3-4158-9171-8320ce9291d4\") "
Oct 30 01:18:51.635845 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:51.635702    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/914501a6-32b3-4158-9171-8320ce9291d4-istio-token\" (UniqueName: \"kubernetes.io/projected/914501a6-32b3-4158-9171-8320ce9291d4-istio-token\") pod \"914501a6-32b3-4158-9171-8320ce9291d4\" (UID: \"914501a6-32b3-4158-9171-8320ce9291d4\") "
Oct 30 01:18:51.635927 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:51.635862    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914501a6-32b3-4158-9171-8320ce9291d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914501a6-32b3-4158-9171-8320ce9291d4-workload-certs\") pod \"914501a6-32b3-4158-9171-8320ce9291d4\" (UID: \"914501a6-32b3-4158-9171-8320ce9291d4\") "
Oct 30 01:18:51.635927 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:51.635863    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914501a6-32b3-4158-9171-8320ce9291d4-workload-socket" pod "914501a6-32b3-4158-9171-8320ce9291d4" (UID: "914501a6-32b3-4158-9171-8320ce9291d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:51.636044 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:51.635971    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/914501a6-32b3-4158-9171-8320ce9291d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914501a6-32b3-4158-9171-8320ce9291d4-istio-podinfo\") pod \"914501a6-32b3-4158-9171-8320ce9291d4\" (UID: \"914501a6-32b3-4158-9171-8320ce9291d4\") "
Oct 30 01:18:51.636044 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:51.636034    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914501a6-32b3-4158-9171-8320ce9291d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914501a6-32b3-4158-9171-8320ce9291d4-istio-data\") pod \"914501a6-32b3-4158-9171-8320ce9291d4\" (UID: \"914501a6-32b3-4158-9171-8320ce9291d4\") "
Oct 30 01:18:51.636160 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:51.636083    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914501a6-32b3-4158-9171-8320ce9291d4-workload-certs" pod "914501a6-32b3-4158-9171-8320ce9291d4" (UID: "914501a6-32b3-4158-9171-8320ce9291d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:51.636160 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:51.636091    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914501a6-32b3-4158-9171-8320ce9291d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914501a6-32b3-4158-9171-8320ce9291d4-credential-socket\") pod \"914501a6-32b3-4158-9171-8320ce9291d4\" (UID: \"914501a6-32b3-4158-9171-8320ce9291d4\") "
Oct 30 01:18:51.636160 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:51.636153    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/914501a6-32b3-4158-9171-8320ce9291d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914501a6-32b3-4158-9171-8320ce9291d4-istiod-ca-cert\") pod \"914501a6-32b3-4158-9171-8320ce9291d4\" (UID: \"914501a6-32b3-4158-9171-8320ce9291d4\") "
Oct 30 01:18:51.636320 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:51.636286    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/914501a6-32b3-4158-9171-8320ce9291d4-kube-api-access-6gnvl\" (UniqueName: \"kubernetes.io/projected/914501a6-32b3-4158-9171-8320ce9291d4-kube-api-access-6gnvl\") pod \"914501a6-32b3-4158-9171-8320ce9291d4\" (UID: \"914501a6-32b3-4158-9171-8320ce9291d4\") "
Oct 30 01:18:51.636374 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:51.636342    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914501a6-32b3-4158-9171-8320ce9291d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914501a6-32b3-4158-9171-8320ce9291d4-istio-envoy\") pod \"914501a6-32b3-4158-9171-8320ce9291d4\" (UID: \"914501a6-32b3-4158-9171-8320ce9291d4\") "
Oct 30 01:18:51.636374 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:51.636281    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914501a6-32b3-4158-9171-8320ce9291d4-istio-data" pod "914501a6-32b3-4158-9171-8320ce9291d4" (UID: "914501a6-32b3-4158-9171-8320ce9291d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:51.636475 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:51.636293    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914501a6-32b3-4158-9171-8320ce9291d4-credential-socket" pod "914501a6-32b3-4158-9171-8320ce9291d4" (UID: "914501a6-32b3-4158-9171-8320ce9291d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:51.636576 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:51.636537    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/914501a6-32b3-4158-9171-8320ce9291d4-istiod-ca-cert" pod "914501a6-32b3-4158-9171-8320ce9291d4" (UID: "914501a6-32b3-4158-9171-8320ce9291d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:18:51.637037 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:51.636955    4393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914501a6-32b3-4158-9171-8320ce9291d4-workload-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:18:51.637037 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:51.636976    4393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914501a6-32b3-4158-9171-8320ce9291d4-istio-data\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:18:51.637037 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:51.636989    4393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914501a6-32b3-4158-9171-8320ce9291d4-credential-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:18:51.637037 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:51.637003    4393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914501a6-32b3-4158-9171-8320ce9291d4-istiod-ca-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:18:51.637037 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:51.637017    4393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914501a6-32b3-4158-9171-8320ce9291d4-workload-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:18:51.638394 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:51.638370    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/914501a6-32b3-4158-9171-8320ce9291d4-istio-podinfo" pod "914501a6-32b3-4158-9171-8320ce9291d4" (UID: "914501a6-32b3-4158-9171-8320ce9291d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:18:51.638753 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:51.638732    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914501a6-32b3-4158-9171-8320ce9291d4-istio-envoy" pod "914501a6-32b3-4158-9171-8320ce9291d4" (UID: "914501a6-32b3-4158-9171-8320ce9291d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:18:51.638931 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:51.638911    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/914501a6-32b3-4158-9171-8320ce9291d4-kube-api-access-6gnvl" pod "914501a6-32b3-4158-9171-8320ce9291d4" (UID: "914501a6-32b3-4158-9171-8320ce9291d4"). InnerVolumeSpecName "kube-api-access-6gnvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:51.638979 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:51.638940    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/914501a6-32b3-4158-9171-8320ce9291d4-istio-token" pod "914501a6-32b3-4158-9171-8320ce9291d4" (UID: "914501a6-32b3-4158-9171-8320ce9291d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:18:51.686023 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:51.685998    4393 scope.go:122] "RemoveContainer" containerID="4b3389fa5dc65f2f7f7e14ea8d839ab90e80abfba04a50a11cb7b2d0b8c01bc0"
Oct 30 01:18:51.697190 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:51.697163    4393 scope.go:122] "RemoveContainer" containerID="ab90a4a48d48ca894dc293ecd89160bd83c83cdfb92c5de4e7b845a90520eba0"
Oct 30 01:18:51.738617 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:51.738416    4393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914501a6-32b3-4158-9171-8320ce9291d4-istio-podinfo\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:18:51.738617 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:51.738448    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gnvl\" (UniqueName: \"kubernetes.io/projected/914501a6-32b3-4158-9171-8320ce9291d4-kube-api-access-6gnvl\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:18:51.738617 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:51.738464    4393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914501a6-32b3-4158-9171-8320ce9291d4-istio-envoy\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:18:51.738617 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:51.738479    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914501a6-32b3-4158-9171-8320ce9291d4-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:18:51.765716 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:51.765686    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"
Oct 30 01:18:51.765716 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:51.765694    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx" event={"ID":"914501a6-32b3-4158-9171-8320ce9291d4","Type":"ContainerDied","Data":"4b3389fa5dc65f2f7f7e14ea8d839ab90e80abfba04a50a11cb7b2d0b8c01bc0"}
Oct 30 01:18:51.765908 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:51.765750    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx" event={"ID":"914501a6-32b3-4158-9171-8320ce9291d4","Type":"ContainerDied","Data":"0878f976cd8531e2040c332d2d313cf9f7dee0a5ddecb28d43daa3b2da5fca48"}
Oct 30 01:18:51.846256 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:51.846217    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"]
Oct 30 01:18:51.850235 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:51.850207    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-29be73-openshift-default-66467f5dcf-7qlrx"]
Oct 30 01:18:53.727014 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:53.726980    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="914501a6-32b3-4158-9171-8320ce9291d4" path="/var/lib/kubelet/pods/914501a6-32b3-4158-9171-8320ce9291d4/volumes"
Oct 30 01:18:55.726471 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:55.726445    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9a8001-646fd774b6-5xz5c"
Oct 30 01:18:55.727042 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:55.726505    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-6f5b8674dc-xch8n"
Oct 30 01:18:56.512758 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:18:56.512721    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:18:58.682077 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:58.682023    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-5f95998c75-v5lq9"]
Oct 30 01:18:58.686393 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:58.686362    4393 cpu_manager.go:414] "containerMap: removing container" podUID="914501a6-32b3-4158-9171-8320ce9291d4" containerName="istio-proxy"
Oct 30 01:18:58.686650 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:58.686628    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="914501a6-32b3-4158-9171-8320ce9291d4" containerName="istio-proxy"
Oct 30 01:18:58.687129 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:58.687108    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="914501a6-32b3-4158-9171-8320ce9291d4" containerName="istio-proxy"
Oct 30 01:18:58.703736 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:58.703694    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-5f95998c75-v5lq9"
Oct 30 01:18:58.736800 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:58.734522    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-812ba2-dockercfg-h2q6l\""
Oct 30 01:18:58.736800 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:58.734650    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-812ba2\""
Oct 30 01:18:58.736800 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:58.735799    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-5f95998c75-v5lq9"]
Oct 30 01:18:58.863481 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:58.863439    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5ckr\" (UniqueName: \"kubernetes.io/projected/be58de6c-a4f3-40e5-9270-689a5eee91f1-kube-api-access-s5ckr\") pod \"payload-pre-processing-e2e-ipp-a-812ba2-5f95998c75-v5lq9\" (UID: \"be58de6c-a4f3-40e5-9270-689a5eee91f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-5f95998c75-v5lq9"
Oct 30 01:18:58.863706 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:58.863676    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be58de6c-a4f3-40e5-9270-689a5eee91f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-812ba2-5f95998c75-v5lq9\" (UID: \"be58de6c-a4f3-40e5-9270-689a5eee91f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-5f95998c75-v5lq9"
Oct 30 01:18:58.965162 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:58.965060    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be58de6c-a4f3-40e5-9270-689a5eee91f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-812ba2-5f95998c75-v5lq9\" (UID: \"be58de6c-a4f3-40e5-9270-689a5eee91f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-5f95998c75-v5lq9"
Oct 30 01:18:58.965345 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:58.965270    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5ckr\" (UniqueName: \"kubernetes.io/projected/be58de6c-a4f3-40e5-9270-689a5eee91f1-kube-api-access-s5ckr\") pod \"payload-pre-processing-e2e-ipp-a-812ba2-5f95998c75-v5lq9\" (UID: \"be58de6c-a4f3-40e5-9270-689a5eee91f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-5f95998c75-v5lq9"
Oct 30 01:18:58.965856 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:58.965834    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be58de6c-a4f3-40e5-9270-689a5eee91f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-812ba2-5f95998c75-v5lq9\" (UID: \"be58de6c-a4f3-40e5-9270-689a5eee91f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-5f95998c75-v5lq9"
Oct 30 01:18:58.988056 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:58.988029    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5ckr\" (UniqueName: \"kubernetes.io/projected/be58de6c-a4f3-40e5-9270-689a5eee91f1-kube-api-access-s5ckr\") pod \"payload-pre-processing-e2e-ipp-a-812ba2-5f95998c75-v5lq9\" (UID: \"be58de6c-a4f3-40e5-9270-689a5eee91f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-5f95998c75-v5lq9"
Oct 30 01:18:59.030947 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:59.030913    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-5f95998c75-v5lq9"
Oct 30 01:18:59.336853 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:18:59.336820    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe58de6c_a4f3_40e5_9270_689a5eee91f1.slice/crio-dcdc97a089365076aaf3051b4fe62158290752af9703fb786f31a9f410fcb62c WatchSource:0}: Error finding container dcdc97a089365076aaf3051b4fe62158290752af9703fb786f31a9f410fcb62c: Status 404 returned error can't find the container with id dcdc97a089365076aaf3051b4fe62158290752af9703fb786f31a9f410fcb62c
Oct 30 01:18:59.337688 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:59.337664    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-5f95998c75-v5lq9"]
Oct 30 01:18:59.831429 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:59.831374    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-5f95998c75-v5lq9" event={"ID":"be58de6c-a4f3-40e5-9270-689a5eee91f1","Type":"ContainerStarted","Data":"1a58dfce66fd7c417df543896afe0ceb6bc1334ded8e5810fa058372f40d35cd"}
Oct 30 01:18:59.831986 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:59.831438    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-5f95998c75-v5lq9" event={"ID":"be58de6c-a4f3-40e5-9270-689a5eee91f1","Type":"ContainerStarted","Data":"dcdc97a089365076aaf3051b4fe62158290752af9703fb786f31a9f410fcb62c"}
Oct 30 01:18:59.831986 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:59.831462    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-5f95998c75-v5lq9"
Oct 30 01:18:59.872675 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:18:59.872615    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-5f95998c75-v5lq9" podStartSLOduration=1.872600083 podStartE2EDuration="1.872600083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:18:59.8699721 +0000 UTC m=+2830.257648492" watchObservedRunningTime="2026-10-30 01:18:59.872600083 +0000 UTC m=+2830.260276471"
Oct 30 01:19:10.840609 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:10.840526    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-5f95998c75-v5lq9"
Oct 30 01:19:26.512652 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:19:26.512612    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:19:36.984472 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:36.984435    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq"]
Oct 30 01:19:36.985010 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:36.984965    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq" podUID="f10a91a2-6575-46ff-a98d-73317cf11bfc" containerName="maas-api" containerID="cri-o://a45d15ab337fe16c1a7cf8bf6948ff118f5e06e3b0b71bdfe25358e8fcdaaafb" gracePeriod=30
Oct 30 01:19:37.130255 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:37.130222    4393 generic.go:356] "Generic (PLEG): container finished" podID="f10a91a2-6575-46ff-a98d-73317cf11bfc" containerID="a45d15ab337fe16c1a7cf8bf6948ff118f5e06e3b0b71bdfe25358e8fcdaaafb" exitCode=0
Oct 30 01:19:37.130417 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:37.130292    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq" event={"ID":"f10a91a2-6575-46ff-a98d-73317cf11bfc","Type":"ContainerDied","Data":"a45d15ab337fe16c1a7cf8bf6948ff118f5e06e3b0b71bdfe25358e8fcdaaafb"}
Oct 30 01:19:37.294987 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:37.294961    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq"
Oct 30 01:19:37.368359 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:37.368325    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f10a91a2-6575-46ff-a98d-73317cf11bfc-kube-api-access-97tjm\" (UniqueName: \"kubernetes.io/projected/f10a91a2-6575-46ff-a98d-73317cf11bfc-kube-api-access-97tjm\") pod \"f10a91a2-6575-46ff-a98d-73317cf11bfc\" (UID: \"f10a91a2-6575-46ff-a98d-73317cf11bfc\") "
Oct 30 01:19:37.368359 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:37.368361    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f10a91a2-6575-46ff-a98d-73317cf11bfc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f10a91a2-6575-46ff-a98d-73317cf11bfc-metrics-certs\") pod \"f10a91a2-6575-46ff-a98d-73317cf11bfc\" (UID: \"f10a91a2-6575-46ff-a98d-73317cf11bfc\") "
Oct 30 01:19:37.368585 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:37.368434    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f10a91a2-6575-46ff-a98d-73317cf11bfc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f10a91a2-6575-46ff-a98d-73317cf11bfc-maas-api-tls\") pod \"f10a91a2-6575-46ff-a98d-73317cf11bfc\" (UID: \"f10a91a2-6575-46ff-a98d-73317cf11bfc\") "
Oct 30 01:19:37.368652 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:37.368606    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f10a91a2-6575-46ff-a98d-73317cf11bfc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f10a91a2-6575-46ff-a98d-73317cf11bfc-openshift-service-ca\") pod \"f10a91a2-6575-46ff-a98d-73317cf11bfc\" (UID: \"f10a91a2-6575-46ff-a98d-73317cf11bfc\") "
Oct 30 01:19:37.369048 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:37.369016    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f10a91a2-6575-46ff-a98d-73317cf11bfc-openshift-service-ca" pod "f10a91a2-6575-46ff-a98d-73317cf11bfc" (UID: "f10a91a2-6575-46ff-a98d-73317cf11bfc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:37.369259 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:37.369202    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f10a91a2-6575-46ff-a98d-73317cf11bfc-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:19:37.371143 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:37.371114    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f10a91a2-6575-46ff-a98d-73317cf11bfc-kube-api-access-97tjm" pod "f10a91a2-6575-46ff-a98d-73317cf11bfc" (UID: "f10a91a2-6575-46ff-a98d-73317cf11bfc"). InnerVolumeSpecName "kube-api-access-97tjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:37.371251 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:37.371164    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f10a91a2-6575-46ff-a98d-73317cf11bfc-maas-api-tls" pod "f10a91a2-6575-46ff-a98d-73317cf11bfc" (UID: "f10a91a2-6575-46ff-a98d-73317cf11bfc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:19:37.371889 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:37.371866    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f10a91a2-6575-46ff-a98d-73317cf11bfc-metrics-certs" pod "f10a91a2-6575-46ff-a98d-73317cf11bfc" (UID: "f10a91a2-6575-46ff-a98d-73317cf11bfc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:19:37.514864 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:37.511432    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97tjm\" (UniqueName: \"kubernetes.io/projected/f10a91a2-6575-46ff-a98d-73317cf11bfc-kube-api-access-97tjm\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:19:37.514864 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:37.511467    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f10a91a2-6575-46ff-a98d-73317cf11bfc-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:19:37.514864 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:37.511485    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f10a91a2-6575-46ff-a98d-73317cf11bfc-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:19:38.138984 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:38.138951    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq"
Oct 30 01:19:38.139447 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:38.138987    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq" event={"ID":"f10a91a2-6575-46ff-a98d-73317cf11bfc","Type":"ContainerDied","Data":"f7046009f8efd6c3bdb69ea0fe68a522617a6b6edbc627a21c524d223012a17c"}
Oct 30 01:19:38.139447 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:38.139045    4393 scope.go:122] "RemoveContainer" containerID="a45d15ab337fe16c1a7cf8bf6948ff118f5e06e3b0b71bdfe25358e8fcdaaafb"
Oct 30 01:19:38.191347 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:38.191315    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq"]
Oct 30 01:19:38.202337 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:38.202312    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a7ca1f-c87fdbbf8-m2wrq"]
Oct 30 01:19:39.726369 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:39.726335    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f10a91a2-6575-46ff-a98d-73317cf11bfc" path="/var/lib/kubelet/pods/f10a91a2-6575-46ff-a98d-73317cf11bfc/volumes"
Oct 30 01:19:43.295580 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:43.295536    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k"]
Oct 30 01:19:43.411688 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:43.411644    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z"]
Oct 30 01:19:43.411974 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:43.411934    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z" podUID="914a8a1b-f13f-4a16-be9d-ebcbda26ac98" containerName="maas-api" containerID="cri-o://a5db9bdc0ca748998b4676d0fe7822e931a5372c96b50b7ece4973c2d2e2af44" gracePeriod=30
Oct 30 01:19:43.564299 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:43.564215    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z" podUID="914a8a1b-f13f-4a16-be9d-ebcbda26ac98" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.185:8443/health\": dial tcp 10.133.0.185:8443: connect: connection refused"
Oct 30 01:19:43.723676 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:43.723649    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z"
Oct 30 01:19:43.816974 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:43.816887    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/914a8a1b-f13f-4a16-be9d-ebcbda26ac98-kube-api-access-sz8qg\" (UniqueName: \"kubernetes.io/projected/914a8a1b-f13f-4a16-be9d-ebcbda26ac98-kube-api-access-sz8qg\") pod \"914a8a1b-f13f-4a16-be9d-ebcbda26ac98\" (UID: \"914a8a1b-f13f-4a16-be9d-ebcbda26ac98\") "
Oct 30 01:19:43.816974 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:43.816965    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/914a8a1b-f13f-4a16-be9d-ebcbda26ac98-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/914a8a1b-f13f-4a16-be9d-ebcbda26ac98-openshift-service-ca\") pod \"914a8a1b-f13f-4a16-be9d-ebcbda26ac98\" (UID: \"914a8a1b-f13f-4a16-be9d-ebcbda26ac98\") "
Oct 30 01:19:43.817226 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:43.817087    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/914a8a1b-f13f-4a16-be9d-ebcbda26ac98-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/914a8a1b-f13f-4a16-be9d-ebcbda26ac98-maas-api-tls\") pod \"914a8a1b-f13f-4a16-be9d-ebcbda26ac98\" (UID: \"914a8a1b-f13f-4a16-be9d-ebcbda26ac98\") "
Oct 30 01:19:43.817294 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:43.817225    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/914a8a1b-f13f-4a16-be9d-ebcbda26ac98-metrics-certs\" (UniqueName: \"kubernetes.io/secret/914a8a1b-f13f-4a16-be9d-ebcbda26ac98-metrics-certs\") pod \"914a8a1b-f13f-4a16-be9d-ebcbda26ac98\" (UID: \"914a8a1b-f13f-4a16-be9d-ebcbda26ac98\") "
Oct 30 01:19:43.817582 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:43.817552    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/914a8a1b-f13f-4a16-be9d-ebcbda26ac98-openshift-service-ca" pod "914a8a1b-f13f-4a16-be9d-ebcbda26ac98" (UID: "914a8a1b-f13f-4a16-be9d-ebcbda26ac98"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:19:43.818152 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:43.818127    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/914a8a1b-f13f-4a16-be9d-ebcbda26ac98-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:19:43.819627 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:43.819598    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/914a8a1b-f13f-4a16-be9d-ebcbda26ac98-maas-api-tls" pod "914a8a1b-f13f-4a16-be9d-ebcbda26ac98" (UID: "914a8a1b-f13f-4a16-be9d-ebcbda26ac98"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:19:43.819627 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:43.819615    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/914a8a1b-f13f-4a16-be9d-ebcbda26ac98-metrics-certs" pod "914a8a1b-f13f-4a16-be9d-ebcbda26ac98" (UID: "914a8a1b-f13f-4a16-be9d-ebcbda26ac98"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:19:43.819940 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:43.819920    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/914a8a1b-f13f-4a16-be9d-ebcbda26ac98-kube-api-access-sz8qg" pod "914a8a1b-f13f-4a16-be9d-ebcbda26ac98" (UID: "914a8a1b-f13f-4a16-be9d-ebcbda26ac98"). InnerVolumeSpecName "kube-api-access-sz8qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:43.931753 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:43.930642    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/914a8a1b-f13f-4a16-be9d-ebcbda26ac98-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:19:43.931753 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:43.930679    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/914a8a1b-f13f-4a16-be9d-ebcbda26ac98-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:19:43.931753 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:43.930694    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz8qg\" (UniqueName: \"kubernetes.io/projected/914a8a1b-f13f-4a16-be9d-ebcbda26ac98-kube-api-access-sz8qg\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:19:44.195903 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:44.195865    4393 generic.go:356] "Generic (PLEG): container finished" podID="914a8a1b-f13f-4a16-be9d-ebcbda26ac98" containerID="a5db9bdc0ca748998b4676d0fe7822e931a5372c96b50b7ece4973c2d2e2af44" exitCode=0
Oct 30 01:19:44.196291 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:44.196068    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z" event={"ID":"914a8a1b-f13f-4a16-be9d-ebcbda26ac98","Type":"ContainerDied","Data":"a5db9bdc0ca748998b4676d0fe7822e931a5372c96b50b7ece4973c2d2e2af44"}
Oct 30 01:19:44.196291 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:44.196114    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z" event={"ID":"914a8a1b-f13f-4a16-be9d-ebcbda26ac98","Type":"ContainerDied","Data":"8389ad2072eaabefae4f1d2edd5fb06f38850228cb3d60d1e7c367c36e9df209"}
Oct 30 01:19:44.196291 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:44.196143    4393 scope.go:122] "RemoveContainer" containerID="a5db9bdc0ca748998b4676d0fe7822e931a5372c96b50b7ece4973c2d2e2af44"
Oct 30 01:19:44.196291 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:44.196169    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z"
Oct 30 01:19:44.208806 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:44.208752    4393 scope.go:122] "RemoveContainer" containerID="a5db9bdc0ca748998b4676d0fe7822e931a5372c96b50b7ece4973c2d2e2af44"
Oct 30 01:19:44.209107 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:19:44.209083    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5db9bdc0ca748998b4676d0fe7822e931a5372c96b50b7ece4973c2d2e2af44\": container with ID starting with a5db9bdc0ca748998b4676d0fe7822e931a5372c96b50b7ece4973c2d2e2af44 not found: ID does not exist" containerID="a5db9bdc0ca748998b4676d0fe7822e931a5372c96b50b7ece4973c2d2e2af44"
Oct 30 01:19:44.209197 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:44.209115    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5db9bdc0ca748998b4676d0fe7822e931a5372c96b50b7ece4973c2d2e2af44"} err="failed to get container status \"a5db9bdc0ca748998b4676d0fe7822e931a5372c96b50b7ece4973c2d2e2af44\": rpc error: code = NotFound desc = could not find container \"a5db9bdc0ca748998b4676d0fe7822e931a5372c96b50b7ece4973c2d2e2af44\": container with ID starting with a5db9bdc0ca748998b4676d0fe7822e931a5372c96b50b7ece4973c2d2e2af44 not found: ID does not exist"
Oct 30 01:19:44.310447 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:44.310412    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z"]
Oct 30 01:19:44.322074 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:44.322045    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-01215f-575bbf6c84-kdj7z"]
Oct 30 01:19:45.726490 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:45.726451    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="914a8a1b-f13f-4a16-be9d-ebcbda26ac98" path="/var/lib/kubelet/pods/914a8a1b-f13f-4a16-be9d-ebcbda26ac98/volumes"
Oct 30 01:19:56.514117 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:19:56.514076    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:19:58.318301 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:58.318257    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k" podUID="f22d339c-0c05-4da1-90d1-062f85e597de" containerName="main" containerID="cri-o://91be60d9888388c45963513bae1c5cd9e0a4366764daca484ba0b8eb67673608" gracePeriod=45
Oct 30 01:19:58.615327 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:58.615299    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k"
Oct 30 01:19:58.673574 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:58.673538    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f22d339c-0c05-4da1-90d1-062f85e597de-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f22d339c-0c05-4da1-90d1-062f85e597de-model-cache\") pod \"f22d339c-0c05-4da1-90d1-062f85e597de\" (UID: \"f22d339c-0c05-4da1-90d1-062f85e597de\") "
Oct 30 01:19:58.673813 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:58.673674    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f22d339c-0c05-4da1-90d1-062f85e597de-home\" (UniqueName: \"kubernetes.io/empty-dir/f22d339c-0c05-4da1-90d1-062f85e597de-home\") pod \"f22d339c-0c05-4da1-90d1-062f85e597de\" (UID: \"f22d339c-0c05-4da1-90d1-062f85e597de\") "
Oct 30 01:19:58.673813 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:58.673739    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f22d339c-0c05-4da1-90d1-062f85e597de-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f22d339c-0c05-4da1-90d1-062f85e597de-dshm\") pod \"f22d339c-0c05-4da1-90d1-062f85e597de\" (UID: \"f22d339c-0c05-4da1-90d1-062f85e597de\") "
Oct 30 01:19:58.673972 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:58.673878    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f22d339c-0c05-4da1-90d1-062f85e597de-kube-api-access-jqlk2\" (UniqueName: \"kubernetes.io/projected/f22d339c-0c05-4da1-90d1-062f85e597de-kube-api-access-jqlk2\") pod \"f22d339c-0c05-4da1-90d1-062f85e597de\" (UID: \"f22d339c-0c05-4da1-90d1-062f85e597de\") "
Oct 30 01:19:58.673972 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:58.673889    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f22d339c-0c05-4da1-90d1-062f85e597de-model-cache" pod "f22d339c-0c05-4da1-90d1-062f85e597de" (UID: "f22d339c-0c05-4da1-90d1-062f85e597de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:58.673972 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:58.673946    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f22d339c-0c05-4da1-90d1-062f85e597de-tls-certs\" (UniqueName: \"kubernetes.io/secret/f22d339c-0c05-4da1-90d1-062f85e597de-tls-certs\") pod \"f22d339c-0c05-4da1-90d1-062f85e597de\" (UID: \"f22d339c-0c05-4da1-90d1-062f85e597de\") "
Oct 30 01:19:58.674168 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:58.673987    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f22d339c-0c05-4da1-90d1-062f85e597de-home" pod "f22d339c-0c05-4da1-90d1-062f85e597de" (UID: "f22d339c-0c05-4da1-90d1-062f85e597de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:58.674168 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:58.674000    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f22d339c-0c05-4da1-90d1-062f85e597de-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f22d339c-0c05-4da1-90d1-062f85e597de-tmp-dir\") pod \"f22d339c-0c05-4da1-90d1-062f85e597de\" (UID: \"f22d339c-0c05-4da1-90d1-062f85e597de\") "
Oct 30 01:19:58.674283 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:58.674212    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f22d339c-0c05-4da1-90d1-062f85e597de-tmp-dir" pod "f22d339c-0c05-4da1-90d1-062f85e597de" (UID: "f22d339c-0c05-4da1-90d1-062f85e597de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:58.675615 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:58.675052    4393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f22d339c-0c05-4da1-90d1-062f85e597de-home\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:19:58.675615 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:58.675079    4393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f22d339c-0c05-4da1-90d1-062f85e597de-tmp-dir\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:19:58.675615 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:58.675119    4393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f22d339c-0c05-4da1-90d1-062f85e597de-model-cache\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:19:58.677164 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:58.677137    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f22d339c-0c05-4da1-90d1-062f85e597de-kube-api-access-jqlk2" pod "f22d339c-0c05-4da1-90d1-062f85e597de" (UID: "f22d339c-0c05-4da1-90d1-062f85e597de"). InnerVolumeSpecName "kube-api-access-jqlk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:19:58.678681 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:58.678646    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f22d339c-0c05-4da1-90d1-062f85e597de-tls-certs" pod "f22d339c-0c05-4da1-90d1-062f85e597de" (UID: "f22d339c-0c05-4da1-90d1-062f85e597de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:19:58.678858 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:58.678830    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f22d339c-0c05-4da1-90d1-062f85e597de-dshm" pod "f22d339c-0c05-4da1-90d1-062f85e597de" (UID: "f22d339c-0c05-4da1-90d1-062f85e597de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:19:58.776728 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:58.776693    4393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f22d339c-0c05-4da1-90d1-062f85e597de-dshm\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:19:58.776728 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:58.776727    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqlk2\" (UniqueName: \"kubernetes.io/projected/f22d339c-0c05-4da1-90d1-062f85e597de-kube-api-access-jqlk2\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:19:58.777006 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:58.776741    4393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f22d339c-0c05-4da1-90d1-062f85e597de-tls-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:19:59.361642 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:59.361608    4393 generic.go:356] "Generic (PLEG): container finished" podID="f22d339c-0c05-4da1-90d1-062f85e597de" containerID="91be60d9888388c45963513bae1c5cd9e0a4366764daca484ba0b8eb67673608" exitCode=0
Oct 30 01:19:59.362107 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:59.361666    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k"
Oct 30 01:19:59.362107 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:59.361685    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k" event={"ID":"f22d339c-0c05-4da1-90d1-062f85e597de","Type":"ContainerDied","Data":"91be60d9888388c45963513bae1c5cd9e0a4366764daca484ba0b8eb67673608"}
Oct 30 01:19:59.362107 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:59.361729    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k" event={"ID":"f22d339c-0c05-4da1-90d1-062f85e597de","Type":"ContainerDied","Data":"4c2b02e240321f14e6aa66a042f06a8ce4c7ac95d3cd988e928558d2b89f2986"}
Oct 30 01:19:59.362107 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:59.361743    4393 scope.go:122] "RemoveContainer" containerID="91be60d9888388c45963513bae1c5cd9e0a4366764daca484ba0b8eb67673608"
Oct 30 01:19:59.373938 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:59.373911    4393 scope.go:122] "RemoveContainer" containerID="91be60d9888388c45963513bae1c5cd9e0a4366764daca484ba0b8eb67673608"
Oct 30 01:19:59.374285 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:19:59.374250    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91be60d9888388c45963513bae1c5cd9e0a4366764daca484ba0b8eb67673608\": container with ID starting with 91be60d9888388c45963513bae1c5cd9e0a4366764daca484ba0b8eb67673608 not found: ID does not exist" containerID="91be60d9888388c45963513bae1c5cd9e0a4366764daca484ba0b8eb67673608"
Oct 30 01:19:59.374378 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:59.374290    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91be60d9888388c45963513bae1c5cd9e0a4366764daca484ba0b8eb67673608"} err="failed to get container status \"91be60d9888388c45963513bae1c5cd9e0a4366764daca484ba0b8eb67673608\": rpc error: code = NotFound desc = could not find container \"91be60d9888388c45963513bae1c5cd9e0a4366764daca484ba0b8eb67673608\": container with ID starting with 91be60d9888388c45963513bae1c5cd9e0a4366764daca484ba0b8eb67673608 not found: ID does not exist"
Oct 30 01:19:59.416028 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:59.415997    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k"]
Oct 30 01:19:59.426722 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:59.426688    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-a7ca1f/ipp-route-a7ca1f-kserve-b6968bd4f-j5g6k"]
Oct 30 01:19:59.726869 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:19:59.726838    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f22d339c-0c05-4da1-90d1-062f85e597de" path="/var/lib/kubelet/pods/f22d339c-0c05-4da1-90d1-062f85e597de/volumes"
Oct 30 01:20:00.326343 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:00.326303    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-16c30f-78c96f8ff4-dwhmx"]
Oct 30 01:20:00.327850 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:00.327825    4393 cpu_manager.go:414] "containerMap: removing container" podUID="914a8a1b-f13f-4a16-be9d-ebcbda26ac98" containerName="maas-api"
Oct 30 01:20:00.328023 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:00.328010    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="914a8a1b-f13f-4a16-be9d-ebcbda26ac98" containerName="maas-api"
Oct 30 01:20:00.328133 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:00.328122    4393 cpu_manager.go:414] "containerMap: removing container" podUID="f22d339c-0c05-4da1-90d1-062f85e597de" containerName="main"
Oct 30 01:20:00.328200 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:00.328191    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f22d339c-0c05-4da1-90d1-062f85e597de" containerName="main"
Oct 30 01:20:00.329529 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:00.329506    4393 cpu_manager.go:414] "containerMap: removing container" podUID="f10a91a2-6575-46ff-a98d-73317cf11bfc" containerName="maas-api"
Oct 30 01:20:00.333852 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:00.329828    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f10a91a2-6575-46ff-a98d-73317cf11bfc" containerName="maas-api"
Oct 30 01:20:00.333852 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:00.333486    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="f10a91a2-6575-46ff-a98d-73317cf11bfc" containerName="maas-api"
Oct 30 01:20:00.333852 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:00.333517    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="914a8a1b-f13f-4a16-be9d-ebcbda26ac98" containerName="maas-api"
Oct 30 01:20:00.333852 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:00.333535    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="f22d339c-0c05-4da1-90d1-062f85e597de" containerName="main"
Oct 30 01:20:00.345421 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:00.345386    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-16c30f-78c96f8ff4-dwhmx"
Oct 30 01:20:00.360022 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:00.359119    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-16c30f-78c96f8ff4-dwhmx"]
Oct 30 01:20:00.366382 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:00.366222    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-16c30f-dockercfg-cn7kp\""
Oct 30 01:20:00.368462 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:00.366273    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-16c30f\""
Oct 30 01:20:00.399740 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:00.399699    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5pcg\" (UniqueName: \"kubernetes.io/projected/de3f97cd-2787-4b28-a883-18c1f36ca325-kube-api-access-j5pcg\") pod \"payload-processing-e2e-ipp-b-16c30f-78c96f8ff4-dwhmx\" (UID: \"de3f97cd-2787-4b28-a883-18c1f36ca325\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-16c30f-78c96f8ff4-dwhmx"
Oct 30 01:20:00.399966 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:00.399766    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de3f97cd-2787-4b28-a883-18c1f36ca325-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-16c30f-78c96f8ff4-dwhmx\" (UID: \"de3f97cd-2787-4b28-a883-18c1f36ca325\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-16c30f-78c96f8ff4-dwhmx"
Oct 30 01:20:00.501083 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:00.501047    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5pcg\" (UniqueName: \"kubernetes.io/projected/de3f97cd-2787-4b28-a883-18c1f36ca325-kube-api-access-j5pcg\") pod \"payload-processing-e2e-ipp-b-16c30f-78c96f8ff4-dwhmx\" (UID: \"de3f97cd-2787-4b28-a883-18c1f36ca325\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-16c30f-78c96f8ff4-dwhmx"
Oct 30 01:20:00.501286 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:00.501097    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de3f97cd-2787-4b28-a883-18c1f36ca325-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-16c30f-78c96f8ff4-dwhmx\" (UID: \"de3f97cd-2787-4b28-a883-18c1f36ca325\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-16c30f-78c96f8ff4-dwhmx"
Oct 30 01:20:00.501872 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:00.501848    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de3f97cd-2787-4b28-a883-18c1f36ca325-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-16c30f-78c96f8ff4-dwhmx\" (UID: \"de3f97cd-2787-4b28-a883-18c1f36ca325\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-16c30f-78c96f8ff4-dwhmx"
Oct 30 01:20:00.524438 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:00.524408    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5pcg\" (UniqueName: \"kubernetes.io/projected/de3f97cd-2787-4b28-a883-18c1f36ca325-kube-api-access-j5pcg\") pod \"payload-processing-e2e-ipp-b-16c30f-78c96f8ff4-dwhmx\" (UID: \"de3f97cd-2787-4b28-a883-18c1f36ca325\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-16c30f-78c96f8ff4-dwhmx"
Oct 30 01:20:00.675082 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:00.675039    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-16c30f-78c96f8ff4-dwhmx"
Oct 30 01:20:00.832325 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:00.832288    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx"]
Oct 30 01:20:00.837243 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:00.837224    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx"
Oct 30 01:20:00.851223 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:00.851191    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-1e8c0e-serving-cert\""
Oct 30 01:20:00.856117 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:00.856084    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx"]
Oct 30 01:20:00.912549 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:00.912510    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a3d613-1b6a-4e50-b5e3-517002e2c5a8-metrics-certs\") pod \"maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx\" (UID: \"56a3d613-1b6a-4e50-b5e3-517002e2c5a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx"
Oct 30 01:20:00.912763 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:00.912743    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56a3d613-1b6a-4e50-b5e3-517002e2c5a8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx\" (UID: \"56a3d613-1b6a-4e50-b5e3-517002e2c5a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx"
Oct 30 01:20:00.912860 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:00.912846    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmkx5\" (UniqueName: \"kubernetes.io/projected/56a3d613-1b6a-4e50-b5e3-517002e2c5a8-kube-api-access-fmkx5\") pod \"maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx\" (UID: \"56a3d613-1b6a-4e50-b5e3-517002e2c5a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx"
Oct 30 01:20:00.912936 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:00.912923    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56a3d613-1b6a-4e50-b5e3-517002e2c5a8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx\" (UID: \"56a3d613-1b6a-4e50-b5e3-517002e2c5a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx"
Oct 30 01:20:00.994267 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:00.994237    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-16c30f-78c96f8ff4-dwhmx"]
Oct 30 01:20:00.994725 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:20:00.994698    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde3f97cd_2787_4b28_a883_18c1f36ca325.slice/crio-e7c9da53cc2f24f15cc3bcfed72b542453762fbad1a99812de18895c5ad8f3be WatchSource:0}: Error finding container e7c9da53cc2f24f15cc3bcfed72b542453762fbad1a99812de18895c5ad8f3be: Status 404 returned error can't find the container with id e7c9da53cc2f24f15cc3bcfed72b542453762fbad1a99812de18895c5ad8f3be
Oct 30 01:20:01.013946 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:01.013909    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56a3d613-1b6a-4e50-b5e3-517002e2c5a8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx\" (UID: \"56a3d613-1b6a-4e50-b5e3-517002e2c5a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx"
Oct 30 01:20:01.014102 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:01.013983    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmkx5\" (UniqueName: \"kubernetes.io/projected/56a3d613-1b6a-4e50-b5e3-517002e2c5a8-kube-api-access-fmkx5\") pod \"maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx\" (UID: \"56a3d613-1b6a-4e50-b5e3-517002e2c5a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx"
Oct 30 01:20:01.014102 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:01.014088    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56a3d613-1b6a-4e50-b5e3-517002e2c5a8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx\" (UID: \"56a3d613-1b6a-4e50-b5e3-517002e2c5a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx"
Oct 30 01:20:01.014220 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:01.014201    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a3d613-1b6a-4e50-b5e3-517002e2c5a8-metrics-certs\") pod \"maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx\" (UID: \"56a3d613-1b6a-4e50-b5e3-517002e2c5a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx"
Oct 30 01:20:01.014839 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:01.014769    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56a3d613-1b6a-4e50-b5e3-517002e2c5a8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx\" (UID: \"56a3d613-1b6a-4e50-b5e3-517002e2c5a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx"
Oct 30 01:20:01.016631 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:01.016608    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56a3d613-1b6a-4e50-b5e3-517002e2c5a8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx\" (UID: \"56a3d613-1b6a-4e50-b5e3-517002e2c5a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx"
Oct 30 01:20:01.016876 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:01.016855    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a3d613-1b6a-4e50-b5e3-517002e2c5a8-metrics-certs\") pod \"maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx\" (UID: \"56a3d613-1b6a-4e50-b5e3-517002e2c5a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx"
Oct 30 01:20:01.037548 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:01.037520    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmkx5\" (UniqueName: \"kubernetes.io/projected/56a3d613-1b6a-4e50-b5e3-517002e2c5a8-kube-api-access-fmkx5\") pod \"maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx\" (UID: \"56a3d613-1b6a-4e50-b5e3-517002e2c5a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx"
Oct 30 01:20:01.154380 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:01.154349    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx"
Oct 30 01:20:01.397143 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:01.397092    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-16c30f-78c96f8ff4-dwhmx" event={"ID":"de3f97cd-2787-4b28-a883-18c1f36ca325","Type":"ContainerStarted","Data":"fa9fbadba700e4e5f09c71eac8c3b73ee09d5b6c84ce91baa1421263a4a194a9"}
Oct 30 01:20:01.397561 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:01.397161    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-16c30f-78c96f8ff4-dwhmx" event={"ID":"de3f97cd-2787-4b28-a883-18c1f36ca325","Type":"ContainerStarted","Data":"e7c9da53cc2f24f15cc3bcfed72b542453762fbad1a99812de18895c5ad8f3be"}
Oct 30 01:20:01.397561 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:01.397190    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-16c30f-78c96f8ff4-dwhmx"
Oct 30 01:20:01.439556 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:01.439489    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-16c30f-78c96f8ff4-dwhmx" podStartSLOduration=1.439468414 podStartE2EDuration="1.439468414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:01.436741208 +0000 UTC m=+2891.824417598" watchObservedRunningTime="2026-10-30 01:20:01.439468414 +0000 UTC m=+2891.827144807"
Oct 30 01:20:01.475552 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:01.475518    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx"]
Oct 30 01:20:01.476141 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:20:01.476116    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a3d613_1b6a_4e50_b5e3_517002e2c5a8.slice/crio-0bf20a73c4b725637330ac85e8da76247ca137398ad928fefc7163953a9913b0 WatchSource:0}: Error finding container 0bf20a73c4b725637330ac85e8da76247ca137398ad928fefc7163953a9913b0: Status 404 returned error can't find the container with id 0bf20a73c4b725637330ac85e8da76247ca137398ad928fefc7163953a9913b0
Oct 30 01:20:02.407665 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:02.407612    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx" event={"ID":"56a3d613-1b6a-4e50-b5e3-517002e2c5a8","Type":"ContainerStarted","Data":"587b916435d155a6010479d1fa562e01ab8ced376a9b0a30512ebc03cf84f082"}
Oct 30 01:20:02.408226 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:02.407681    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx" event={"ID":"56a3d613-1b6a-4e50-b5e3-517002e2c5a8","Type":"ContainerStarted","Data":"0bf20a73c4b725637330ac85e8da76247ca137398ad928fefc7163953a9913b0"}
Oct 30 01:20:02.408226 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:02.407711    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx"
Oct 30 01:20:02.448565 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:02.448504    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx" podStartSLOduration=2.082239334 podStartE2EDuration="2.448490551s" totalImagesPullingTime="366.251217ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:02.447611424 +0000 UTC m=+2892.835287813" watchObservedRunningTime="2026-10-30 01:20:02.448490551 +0000 UTC m=+2892.836166978"
Oct 30 01:20:08.419896 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:08.419869    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx"
Oct 30 01:20:11.260526 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:11.252220    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc"]
Oct 30 01:20:11.261566 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:11.261493    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc" podUID="25aa52b6-2bb8-457c-9708-d4147aed2544" containerName="maas-api" containerID="cri-o://b5c6ee3766c230cb0f0f5d64679aeb70c14aaaaf333c1707f6e6830cd3a90f80" gracePeriod=30
Oct 30 01:20:11.421399 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:11.421358    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9a8001-646fd774b6-5xz5c"]
Oct 30 01:20:11.421655 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:11.421608    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-9a8001-646fd774b6-5xz5c" podUID="5428a426-9be0-41b6-b518-beaa791a21c7" containerName="payload-processing" containerID="cri-o://2b6799735d132e44379e505e4384369d1592a910410879ec0eede729925ef1b5" gracePeriod=30
Oct 30 01:20:11.442450 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:11.442396    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-6f5b8674dc-xch8n"]
Oct 30 01:20:11.443017 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:11.442963    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-6f5b8674dc-xch8n" podUID="5b80809c-0baa-4d5c-81c5-ea61440aadff" containerName="payload-pre-processing" containerID="cri-o://dd01657c7300c76c844b71398d2352fa5f6baa2d9844fdc5584a3371cce9f822" gracePeriod=30
Oct 30 01:20:11.514880 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:11.506869    4393 generic.go:356] "Generic (PLEG): container finished" podID="25aa52b6-2bb8-457c-9708-d4147aed2544" containerID="b5c6ee3766c230cb0f0f5d64679aeb70c14aaaaf333c1707f6e6830cd3a90f80" exitCode=0
Oct 30 01:20:11.514880 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:11.507038    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc" event={"ID":"25aa52b6-2bb8-457c-9708-d4147aed2544","Type":"ContainerDied","Data":"b5c6ee3766c230cb0f0f5d64679aeb70c14aaaaf333c1707f6e6830cd3a90f80"}
Oct 30 01:20:11.625916 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:11.625885    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc"
Oct 30 01:20:11.727251 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:11.725239    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25aa52b6-2bb8-457c-9708-d4147aed2544-metrics-certs\" (UniqueName: \"kubernetes.io/secret/25aa52b6-2bb8-457c-9708-d4147aed2544-metrics-certs\") pod \"25aa52b6-2bb8-457c-9708-d4147aed2544\" (UID: \"25aa52b6-2bb8-457c-9708-d4147aed2544\") "
Oct 30 01:20:11.727251 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:11.725434    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25aa52b6-2bb8-457c-9708-d4147aed2544-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25aa52b6-2bb8-457c-9708-d4147aed2544-maas-api-tls\") pod \"25aa52b6-2bb8-457c-9708-d4147aed2544\" (UID: \"25aa52b6-2bb8-457c-9708-d4147aed2544\") "
Oct 30 01:20:11.727914 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:11.727876    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25aa52b6-2bb8-457c-9708-d4147aed2544-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25aa52b6-2bb8-457c-9708-d4147aed2544-openshift-service-ca\") pod \"25aa52b6-2bb8-457c-9708-d4147aed2544\" (UID: \"25aa52b6-2bb8-457c-9708-d4147aed2544\") "
Oct 30 01:20:11.728173 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:11.728146    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25aa52b6-2bb8-457c-9708-d4147aed2544-kube-api-access-tlmnh\" (UniqueName: \"kubernetes.io/projected/25aa52b6-2bb8-457c-9708-d4147aed2544-kube-api-access-tlmnh\") pod \"25aa52b6-2bb8-457c-9708-d4147aed2544\" (UID: \"25aa52b6-2bb8-457c-9708-d4147aed2544\") "
Oct 30 01:20:11.734651 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:11.734579    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25aa52b6-2bb8-457c-9708-d4147aed2544-maas-api-tls" pod "25aa52b6-2bb8-457c-9708-d4147aed2544" (UID: "25aa52b6-2bb8-457c-9708-d4147aed2544"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:20:11.740017 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:11.739959    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25aa52b6-2bb8-457c-9708-d4147aed2544-kube-api-access-tlmnh" pod "25aa52b6-2bb8-457c-9708-d4147aed2544" (UID: "25aa52b6-2bb8-457c-9708-d4147aed2544"). InnerVolumeSpecName "kube-api-access-tlmnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:11.742102 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:11.742063    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25aa52b6-2bb8-457c-9708-d4147aed2544-openshift-service-ca" pod "25aa52b6-2bb8-457c-9708-d4147aed2544" (UID: "25aa52b6-2bb8-457c-9708-d4147aed2544"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:11.742258 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:11.742186    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25aa52b6-2bb8-457c-9708-d4147aed2544-metrics-certs" pod "25aa52b6-2bb8-457c-9708-d4147aed2544" (UID: "25aa52b6-2bb8-457c-9708-d4147aed2544"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:20:11.781453 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:11.781414    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9a8001-646fd774b6-5xz5c"
Oct 30 01:20:11.822806 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:11.818449    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-6f5b8674dc-xch8n"
Oct 30 01:20:11.849851 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:11.849253    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5428a426-9be0-41b6-b518-beaa791a21c7-kube-api-access-9v7lf\" (UniqueName: \"kubernetes.io/projected/5428a426-9be0-41b6-b518-beaa791a21c7-kube-api-access-9v7lf\") pod \"5428a426-9be0-41b6-b518-beaa791a21c7\" (UID: \"5428a426-9be0-41b6-b518-beaa791a21c7\") "
Oct 30 01:20:11.849851 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:11.849407    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b80809c-0baa-4d5c-81c5-ea61440aadff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b80809c-0baa-4d5c-81c5-ea61440aadff-plugins-config-volume\") pod \"5b80809c-0baa-4d5c-81c5-ea61440aadff\" (UID: \"5b80809c-0baa-4d5c-81c5-ea61440aadff\") "
Oct 30 01:20:11.849851 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:11.849608    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b80809c-0baa-4d5c-81c5-ea61440aadff-kube-api-access-9q628\" (UniqueName: \"kubernetes.io/projected/5b80809c-0baa-4d5c-81c5-ea61440aadff-kube-api-access-9q628\") pod \"5b80809c-0baa-4d5c-81c5-ea61440aadff\" (UID: \"5b80809c-0baa-4d5c-81c5-ea61440aadff\") "
Oct 30 01:20:11.849851 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:11.849691    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5428a426-9be0-41b6-b518-beaa791a21c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5428a426-9be0-41b6-b518-beaa791a21c7-plugins-config-volume\") pod \"5428a426-9be0-41b6-b518-beaa791a21c7\" (UID: \"5428a426-9be0-41b6-b518-beaa791a21c7\") "
Oct 30 01:20:11.878805 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:11.850472    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlmnh\" (UniqueName: \"kubernetes.io/projected/25aa52b6-2bb8-457c-9708-d4147aed2544-kube-api-access-tlmnh\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:20:11.878805 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:11.850514    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25aa52b6-2bb8-457c-9708-d4147aed2544-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:20:11.878805 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:11.850533    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25aa52b6-2bb8-457c-9708-d4147aed2544-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:20:11.878805 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:11.850553    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25aa52b6-2bb8-457c-9708-d4147aed2544-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:20:11.878805 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:11.863662    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5428a426-9be0-41b6-b518-beaa791a21c7-plugins-config-volume" pod "5428a426-9be0-41b6-b518-beaa791a21c7" (UID: "5428a426-9be0-41b6-b518-beaa791a21c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:11.878805 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:11.863970    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b80809c-0baa-4d5c-81c5-ea61440aadff-plugins-config-volume" pod "5b80809c-0baa-4d5c-81c5-ea61440aadff" (UID: "5b80809c-0baa-4d5c-81c5-ea61440aadff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:20:11.930524 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:11.930458    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5428a426-9be0-41b6-b518-beaa791a21c7-kube-api-access-9v7lf" pod "5428a426-9be0-41b6-b518-beaa791a21c7" (UID: "5428a426-9be0-41b6-b518-beaa791a21c7"). InnerVolumeSpecName "kube-api-access-9v7lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:11.933494 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:11.933463    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b80809c-0baa-4d5c-81c5-ea61440aadff-kube-api-access-9q628" pod "5b80809c-0baa-4d5c-81c5-ea61440aadff" (UID: "5b80809c-0baa-4d5c-81c5-ea61440aadff"). InnerVolumeSpecName "kube-api-access-9q628". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:20:11.953237 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:11.953140    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v7lf\" (UniqueName: \"kubernetes.io/projected/5428a426-9be0-41b6-b518-beaa791a21c7-kube-api-access-9v7lf\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:20:11.953237 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:11.953179    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b80809c-0baa-4d5c-81c5-ea61440aadff-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:20:11.953237 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:11.953195    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q628\" (UniqueName: \"kubernetes.io/projected/5b80809c-0baa-4d5c-81c5-ea61440aadff-kube-api-access-9q628\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:20:11.953237 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:11.953210    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5428a426-9be0-41b6-b518-beaa791a21c7-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:20:12.414880 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:12.413307    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-16c30f-78c96f8ff4-dwhmx"
Oct 30 01:20:12.517178 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:12.517140    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc"
Oct 30 01:20:12.517388 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:12.517169    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc" event={"ID":"25aa52b6-2bb8-457c-9708-d4147aed2544","Type":"ContainerDied","Data":"1cd5e196754ac806cf966fbab8461db54b35ce8dddd0aed1ff8299dd0ffb0705"}
Oct 30 01:20:12.517388 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:12.517249    4393 scope.go:122] "RemoveContainer" containerID="b5c6ee3766c230cb0f0f5d64679aeb70c14aaaaf333c1707f6e6830cd3a90f80"
Oct 30 01:20:12.519394 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:12.519343    4393 generic.go:356] "Generic (PLEG): container finished" podID="5b80809c-0baa-4d5c-81c5-ea61440aadff" containerID="dd01657c7300c76c844b71398d2352fa5f6baa2d9844fdc5584a3371cce9f822" exitCode=0
Oct 30 01:20:12.519689 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:12.519410    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-6f5b8674dc-xch8n"
Oct 30 01:20:12.519689 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:12.519404    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-6f5b8674dc-xch8n" event={"ID":"5b80809c-0baa-4d5c-81c5-ea61440aadff","Type":"ContainerDied","Data":"dd01657c7300c76c844b71398d2352fa5f6baa2d9844fdc5584a3371cce9f822"}
Oct 30 01:20:12.519689 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:12.519564    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-6f5b8674dc-xch8n" event={"ID":"5b80809c-0baa-4d5c-81c5-ea61440aadff","Type":"ContainerDied","Data":"b996858df314b07b68301964ef4f59b3dda82662cefcf1cb0227eba8250bf7d2"}
Oct 30 01:20:12.521839 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:12.521765    4393 generic.go:356] "Generic (PLEG): container finished" podID="5428a426-9be0-41b6-b518-beaa791a21c7" containerID="2b6799735d132e44379e505e4384369d1592a910410879ec0eede729925ef1b5" exitCode=0
Oct 30 01:20:12.521980 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:12.521965    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9a8001-646fd774b6-5xz5c" event={"ID":"5428a426-9be0-41b6-b518-beaa791a21c7","Type":"ContainerDied","Data":"2b6799735d132e44379e505e4384369d1592a910410879ec0eede729925ef1b5"}
Oct 30 01:20:12.522053 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:12.522001    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9a8001-646fd774b6-5xz5c" event={"ID":"5428a426-9be0-41b6-b518-beaa791a21c7","Type":"ContainerDied","Data":"8a2ae59bc73452d245fdf99e72da19bce15db08b0f91011160e0b336ccb2f8ea"}
Oct 30 01:20:12.522113 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:12.522063    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9a8001-646fd774b6-5xz5c"
Oct 30 01:20:12.529638 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:12.529614    4393 scope.go:122] "RemoveContainer" containerID="dd01657c7300c76c844b71398d2352fa5f6baa2d9844fdc5584a3371cce9f822"
Oct 30 01:20:12.541389 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:12.541360    4393 scope.go:122] "RemoveContainer" containerID="dd01657c7300c76c844b71398d2352fa5f6baa2d9844fdc5584a3371cce9f822"
Oct 30 01:20:12.541739 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:20:12.541716    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd01657c7300c76c844b71398d2352fa5f6baa2d9844fdc5584a3371cce9f822\": container with ID starting with dd01657c7300c76c844b71398d2352fa5f6baa2d9844fdc5584a3371cce9f822 not found: ID does not exist" containerID="dd01657c7300c76c844b71398d2352fa5f6baa2d9844fdc5584a3371cce9f822"
Oct 30 01:20:12.541815 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:12.541751    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd01657c7300c76c844b71398d2352fa5f6baa2d9844fdc5584a3371cce9f822"} err="failed to get container status \"dd01657c7300c76c844b71398d2352fa5f6baa2d9844fdc5584a3371cce9f822\": rpc error: code = NotFound desc = could not find container \"dd01657c7300c76c844b71398d2352fa5f6baa2d9844fdc5584a3371cce9f822\": container with ID starting with dd01657c7300c76c844b71398d2352fa5f6baa2d9844fdc5584a3371cce9f822 not found: ID does not exist"
Oct 30 01:20:12.541815 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:12.541772    4393 scope.go:122] "RemoveContainer" containerID="2b6799735d132e44379e505e4384369d1592a910410879ec0eede729925ef1b5"
Oct 30 01:20:12.554348 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:12.554312    4393 scope.go:122] "RemoveContainer" containerID="2b6799735d132e44379e505e4384369d1592a910410879ec0eede729925ef1b5"
Oct 30 01:20:12.554756 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:20:12.554713    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b6799735d132e44379e505e4384369d1592a910410879ec0eede729925ef1b5\": container with ID starting with 2b6799735d132e44379e505e4384369d1592a910410879ec0eede729925ef1b5 not found: ID does not exist" containerID="2b6799735d132e44379e505e4384369d1592a910410879ec0eede729925ef1b5"
Oct 30 01:20:12.554866 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:12.554764    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b6799735d132e44379e505e4384369d1592a910410879ec0eede729925ef1b5"} err="failed to get container status \"2b6799735d132e44379e505e4384369d1592a910410879ec0eede729925ef1b5\": rpc error: code = NotFound desc = could not find container \"2b6799735d132e44379e505e4384369d1592a910410879ec0eede729925ef1b5\": container with ID starting with 2b6799735d132e44379e505e4384369d1592a910410879ec0eede729925ef1b5 not found: ID does not exist"
Oct 30 01:20:12.598242 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:12.598144    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc"]
Oct 30 01:20:12.602454 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:12.602420    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9a8001-df4bd89bc-48wdc"]
Oct 30 01:20:12.636168 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:12.636137    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-6f5b8674dc-xch8n"]
Oct 30 01:20:12.640408 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:12.640356    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9a8001-6f5b8674dc-xch8n"]
Oct 30 01:20:12.673121 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:12.673082    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9a8001-646fd774b6-5xz5c"]
Oct 30 01:20:12.677248 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:12.677218    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9a8001-646fd774b6-5xz5c"]
Oct 30 01:20:13.727217 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:13.727182    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25aa52b6-2bb8-457c-9708-d4147aed2544" path="/var/lib/kubelet/pods/25aa52b6-2bb8-457c-9708-d4147aed2544/volumes"
Oct 30 01:20:13.727884 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:13.727725    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5428a426-9be0-41b6-b518-beaa791a21c7" path="/var/lib/kubelet/pods/5428a426-9be0-41b6-b518-beaa791a21c7/volumes"
Oct 30 01:20:13.728173 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:13.728157    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b80809c-0baa-4d5c-81c5-ea61440aadff" path="/var/lib/kubelet/pods/5b80809c-0baa-4d5c-81c5-ea61440aadff/volumes"
Oct 30 01:20:18.820560 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:18.820497    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-l4gst_9217cef0-6bf2-41d5-b303-4c4b78dcb7f0/payload-processing/0.log"
Oct 30 01:20:19.010024 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:19.009994    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-l4gst_9217cef0-6bf2-41d5-b303-4c4b78dcb7f0/payload-processing/0.log"
Oct 30 01:20:19.217308 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:19.217269    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-16c30f-78c96f8ff4-dwhmx_de3f97cd-2787-4b28-a883-18c1f36ca325/payload-processing/0.log"
Oct 30 01:20:26.514106 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:20:26.514065    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:20:51.166415 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:51.166378    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5"]
Oct 30 01:20:51.167756 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:51.167729    4393 cpu_manager.go:414] "containerMap: removing container" podUID="25aa52b6-2bb8-457c-9708-d4147aed2544" containerName="maas-api"
Oct 30 01:20:51.167756 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:51.167757    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25aa52b6-2bb8-457c-9708-d4147aed2544" containerName="maas-api"
Oct 30 01:20:51.168019 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:51.168003    4393 cpu_manager.go:414] "containerMap: removing container" podUID="5428a426-9be0-41b6-b518-beaa791a21c7" containerName="payload-processing"
Oct 30 01:20:51.168076 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:51.168020    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5428a426-9be0-41b6-b518-beaa791a21c7" containerName="payload-processing"
Oct 30 01:20:51.168118 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:51.168083    4393 cpu_manager.go:414] "containerMap: removing container" podUID="5b80809c-0baa-4d5c-81c5-ea61440aadff" containerName="payload-pre-processing"
Oct 30 01:20:51.168118 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:51.168095    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b80809c-0baa-4d5c-81c5-ea61440aadff" containerName="payload-pre-processing"
Oct 30 01:20:51.168313 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:51.168299    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="25aa52b6-2bb8-457c-9708-d4147aed2544" containerName="maas-api"
Oct 30 01:20:51.168348 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:51.168325    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="5428a426-9be0-41b6-b518-beaa791a21c7" containerName="payload-processing"
Oct 30 01:20:51.168348 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:51.168340    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b80809c-0baa-4d5c-81c5-ea61440aadff" containerName="payload-pre-processing"
Oct 30 01:20:51.173011 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:51.172985    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5"
Oct 30 01:20:51.191113 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:51.191076    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-658c99-serving-cert\""
Oct 30 01:20:51.204154 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:51.204118    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5"]
Oct 30 01:20:51.282370 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:51.282325    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63d6236e-0643-4747-b66d-bbdee696584d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5\" (UID: \"63d6236e-0643-4747-b66d-bbdee696584d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5"
Oct 30 01:20:51.282607 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:51.282544    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63d6236e-0643-4747-b66d-bbdee696584d-metrics-certs\") pod \"maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5\" (UID: \"63d6236e-0643-4747-b66d-bbdee696584d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5"
Oct 30 01:20:51.282678 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:51.282648    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68pdj\" (UniqueName: \"kubernetes.io/projected/63d6236e-0643-4747-b66d-bbdee696584d-kube-api-access-68pdj\") pod \"maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5\" (UID: \"63d6236e-0643-4747-b66d-bbdee696584d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5"
Oct 30 01:20:51.282807 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:51.282766    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63d6236e-0643-4747-b66d-bbdee696584d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5\" (UID: \"63d6236e-0643-4747-b66d-bbdee696584d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5"
Oct 30 01:20:51.396898 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:51.390989    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63d6236e-0643-4747-b66d-bbdee696584d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5\" (UID: \"63d6236e-0643-4747-b66d-bbdee696584d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5"
Oct 30 01:20:51.396898 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:51.391192    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63d6236e-0643-4747-b66d-bbdee696584d-metrics-certs\") pod \"maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5\" (UID: \"63d6236e-0643-4747-b66d-bbdee696584d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5"
Oct 30 01:20:51.396898 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:51.391254    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68pdj\" (UniqueName: \"kubernetes.io/projected/63d6236e-0643-4747-b66d-bbdee696584d-kube-api-access-68pdj\") pod \"maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5\" (UID: \"63d6236e-0643-4747-b66d-bbdee696584d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5"
Oct 30 01:20:51.396898 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:51.391356    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63d6236e-0643-4747-b66d-bbdee696584d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5\" (UID: \"63d6236e-0643-4747-b66d-bbdee696584d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5"
Oct 30 01:20:51.396898 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:51.392621    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63d6236e-0643-4747-b66d-bbdee696584d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5\" (UID: \"63d6236e-0643-4747-b66d-bbdee696584d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5"
Oct 30 01:20:51.396898 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:20:51.392742    4393 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-658c99-serving-cert: secret "maas-api-e2e-ipp-b-658c99-serving-cert" not found
Oct 30 01:20:51.396898 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:20:51.392838    4393 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63d6236e-0643-4747-b66d-bbdee696584d-maas-api-tls podName:63d6236e-0643-4747-b66d-bbdee696584d nodeName:}" failed. No retries permitted until 2026-10-30 01:20:51.892815406 +0000 UTC m=+2942.280491791 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/63d6236e-0643-4747-b66d-bbdee696584d-maas-api-tls") pod "maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5" (UID: "63d6236e-0643-4747-b66d-bbdee696584d") : secret "maas-api-e2e-ipp-b-658c99-serving-cert" not found
Oct 30 01:20:51.402853 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:51.402737    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63d6236e-0643-4747-b66d-bbdee696584d-metrics-certs\") pod \"maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5\" (UID: \"63d6236e-0643-4747-b66d-bbdee696584d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5"
Oct 30 01:20:51.419043 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:51.418948    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68pdj\" (UniqueName: \"kubernetes.io/projected/63d6236e-0643-4747-b66d-bbdee696584d-kube-api-access-68pdj\") pod \"maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5\" (UID: \"63d6236e-0643-4747-b66d-bbdee696584d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5"
Oct 30 01:20:51.900527 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:51.900481    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63d6236e-0643-4747-b66d-bbdee696584d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5\" (UID: \"63d6236e-0643-4747-b66d-bbdee696584d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5"
Oct 30 01:20:51.903258 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:51.903228    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63d6236e-0643-4747-b66d-bbdee696584d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5\" (UID: \"63d6236e-0643-4747-b66d-bbdee696584d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5"
Oct 30 01:20:52.091818 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:52.091696    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5"
Oct 30 01:20:52.404990 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:52.404951    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5"]
Oct 30 01:20:52.405455 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:20:52.405088    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d6236e_0643_4747_b66d_bbdee696584d.slice/crio-d77c87bbe8212c141b7f6f5d56b626399976671f72e942655b6eb8aa15d9d49d WatchSource:0}: Error finding container d77c87bbe8212c141b7f6f5d56b626399976671f72e942655b6eb8aa15d9d49d: Status 404 returned error can't find the container with id d77c87bbe8212c141b7f6f5d56b626399976671f72e942655b6eb8aa15d9d49d
Oct 30 01:20:52.957167 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:52.957118    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5" event={"ID":"63d6236e-0643-4747-b66d-bbdee696584d","Type":"ContainerStarted","Data":"440c7d49ab41e80459df7be5b76f3bfc6d24dd8541d209825f00bef135563570"}
Oct 30 01:20:52.957378 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:52.957181    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5" event={"ID":"63d6236e-0643-4747-b66d-bbdee696584d","Type":"ContainerStarted","Data":"d77c87bbe8212c141b7f6f5d56b626399976671f72e942655b6eb8aa15d9d49d"}
Oct 30 01:20:52.957378 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:52.957204    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5"
Oct 30 01:20:53.009812 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:53.009257    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5" podStartSLOduration=1.628668147 podStartE2EDuration="2.009232382s" totalImagesPullingTime="380.564235ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:20:52.999887369 +0000 UTC m=+2943.387563760" watchObservedRunningTime="2026-10-30 01:20:53.009232382 +0000 UTC m=+2943.396908772"
Oct 30 01:20:56.513116 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:20:56.513074    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:20:58.970986 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:20:58.970951    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5"
Oct 30 01:21:00.900129 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:00.900097    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"]
Oct 30 01:21:00.900825 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:00.900378    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72" podUID="d5355e22-6596-4c63-a492-fe8b578f04b5" containerName="istio-proxy" containerID="cri-o://12102c58a31179f367c4e9cb084c71ff9b233fd21edbcbe33ad1c0090a4d147e" gracePeriod=30
Oct 30 01:21:01.957706 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:01.957667    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2"]
Oct 30 01:21:01.958170 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:01.957913    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2" podUID="a002f857-7a6e-4fb6-a05c-bce70bac9342" containerName="maas-api" containerID="cri-o://88802c51b03f92250f16d66397ea6fbe94f697e6e44b3ae4ab28e29b552d4e9f" gracePeriod=30
Oct 30 01:21:02.114182 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:02.114141    4393 generic.go:356] "Generic (PLEG): container finished" podID="a002f857-7a6e-4fb6-a05c-bce70bac9342" containerID="88802c51b03f92250f16d66397ea6fbe94f697e6e44b3ae4ab28e29b552d4e9f" exitCode=0
Oct 30 01:21:02.114399 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:02.114272    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2" event={"ID":"a002f857-7a6e-4fb6-a05c-bce70bac9342","Type":"ContainerDied","Data":"88802c51b03f92250f16d66397ea6fbe94f697e6e44b3ae4ab28e29b552d4e9f"}
Oct 30 01:21:02.241999 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:02.241967    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2"
Oct 30 01:21:02.276335 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:02.276301    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a002f857-7a6e-4fb6-a05c-bce70bac9342-kube-api-access-trphh\" (UniqueName: \"kubernetes.io/projected/a002f857-7a6e-4fb6-a05c-bce70bac9342-kube-api-access-trphh\") pod \"a002f857-7a6e-4fb6-a05c-bce70bac9342\" (UID: \"a002f857-7a6e-4fb6-a05c-bce70bac9342\") "
Oct 30 01:21:02.276539 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:02.276354    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a002f857-7a6e-4fb6-a05c-bce70bac9342-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a002f857-7a6e-4fb6-a05c-bce70bac9342-openshift-service-ca\") pod \"a002f857-7a6e-4fb6-a05c-bce70bac9342\" (UID: \"a002f857-7a6e-4fb6-a05c-bce70bac9342\") "
Oct 30 01:21:02.276539 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:02.276392    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a002f857-7a6e-4fb6-a05c-bce70bac9342-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a002f857-7a6e-4fb6-a05c-bce70bac9342-maas-api-tls\") pod \"a002f857-7a6e-4fb6-a05c-bce70bac9342\" (UID: \"a002f857-7a6e-4fb6-a05c-bce70bac9342\") "
Oct 30 01:21:02.276667 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:02.276631    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a002f857-7a6e-4fb6-a05c-bce70bac9342-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a002f857-7a6e-4fb6-a05c-bce70bac9342-metrics-certs\") pod \"a002f857-7a6e-4fb6-a05c-bce70bac9342\" (UID: \"a002f857-7a6e-4fb6-a05c-bce70bac9342\") "
Oct 30 01:21:02.276848 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:02.276820    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a002f857-7a6e-4fb6-a05c-bce70bac9342-openshift-service-ca" pod "a002f857-7a6e-4fb6-a05c-bce70bac9342" (UID: "a002f857-7a6e-4fb6-a05c-bce70bac9342"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:02.277297 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:02.277272    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a002f857-7a6e-4fb6-a05c-bce70bac9342-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:02.279017 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:02.278970    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a002f857-7a6e-4fb6-a05c-bce70bac9342-metrics-certs" pod "a002f857-7a6e-4fb6-a05c-bce70bac9342" (UID: "a002f857-7a6e-4fb6-a05c-bce70bac9342"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:21:02.279366 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:02.279338    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a002f857-7a6e-4fb6-a05c-bce70bac9342-maas-api-tls" pod "a002f857-7a6e-4fb6-a05c-bce70bac9342" (UID: "a002f857-7a6e-4fb6-a05c-bce70bac9342"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:21:02.279426 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:02.279355    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a002f857-7a6e-4fb6-a05c-bce70bac9342-kube-api-access-trphh" pod "a002f857-7a6e-4fb6-a05c-bce70bac9342" (UID: "a002f857-7a6e-4fb6-a05c-bce70bac9342"). InnerVolumeSpecName "kube-api-access-trphh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:02.378676 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:02.378635    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trphh\" (UniqueName: \"kubernetes.io/projected/a002f857-7a6e-4fb6-a05c-bce70bac9342-kube-api-access-trphh\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:02.378676 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:02.378669    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a002f857-7a6e-4fb6-a05c-bce70bac9342-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:02.378676 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:02.378678    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a002f857-7a6e-4fb6-a05c-bce70bac9342-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:03.124062 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:03.124029    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2"
Oct 30 01:21:03.124585 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:03.124065    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2" event={"ID":"a002f857-7a6e-4fb6-a05c-bce70bac9342","Type":"ContainerDied","Data":"1f6ccd7c8f476269e7a3f35826bba510674c50d62d9f2479c2c57b140a8619f8"}
Oct 30 01:21:03.124585 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:03.124133    4393 scope.go:122] "RemoveContainer" containerID="88802c51b03f92250f16d66397ea6fbe94f697e6e44b3ae4ab28e29b552d4e9f"
Oct 30 01:21:03.180106 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:03.180073    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2"]
Oct 30 01:21:03.190533 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:03.190499    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6d5b8-569fbc5c6d-m6vt2"]
Oct 30 01:21:03.726185 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:03.726144    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a002f857-7a6e-4fb6-a05c-bce70bac9342" path="/var/lib/kubelet/pods/a002f857-7a6e-4fb6-a05c-bce70bac9342/volumes"
Oct 30 01:21:06.155296 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:06.155258    4393 generic.go:356] "Generic (PLEG): container finished" podID="d5355e22-6596-4c63-a492-fe8b578f04b5" containerID="12102c58a31179f367c4e9cb084c71ff9b233fd21edbcbe33ad1c0090a4d147e" exitCode=0
Oct 30 01:21:06.156071 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:06.155306    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72" event={"ID":"d5355e22-6596-4c63-a492-fe8b578f04b5","Type":"ContainerDied","Data":"12102c58a31179f367c4e9cb084c71ff9b233fd21edbcbe33ad1c0090a4d147e"}
Oct 30 01:21:06.184981 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:06.184912    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"
Oct 30 01:21:06.232311 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:06.232268    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5355e22-6596-4c63-a492-fe8b578f04b5-istio-token\" (UniqueName: \"kubernetes.io/projected/d5355e22-6596-4c63-a492-fe8b578f04b5-istio-token\") pod \"d5355e22-6596-4c63-a492-fe8b578f04b5\" (UID: \"d5355e22-6596-4c63-a492-fe8b578f04b5\") "
Oct 30 01:21:06.232542 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:06.232367    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5355e22-6596-4c63-a492-fe8b578f04b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5355e22-6596-4c63-a492-fe8b578f04b5-credential-socket\") pod \"d5355e22-6596-4c63-a492-fe8b578f04b5\" (UID: \"d5355e22-6596-4c63-a492-fe8b578f04b5\") "
Oct 30 01:21:06.232542 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:06.232420    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5355e22-6596-4c63-a492-fe8b578f04b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5355e22-6596-4c63-a492-fe8b578f04b5-workload-certs\") pod \"d5355e22-6596-4c63-a492-fe8b578f04b5\" (UID: \"d5355e22-6596-4c63-a492-fe8b578f04b5\") "
Oct 30 01:21:06.232542 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:06.232484    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5355e22-6596-4c63-a492-fe8b578f04b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5355e22-6596-4c63-a492-fe8b578f04b5-istio-envoy\") pod \"d5355e22-6596-4c63-a492-fe8b578f04b5\" (UID: \"d5355e22-6596-4c63-a492-fe8b578f04b5\") "
Oct 30 01:21:06.232733 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:06.232613    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5355e22-6596-4c63-a492-fe8b578f04b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5355e22-6596-4c63-a492-fe8b578f04b5-workload-socket\") pod \"d5355e22-6596-4c63-a492-fe8b578f04b5\" (UID: \"d5355e22-6596-4c63-a492-fe8b578f04b5\") "
Oct 30 01:21:06.232733 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:06.232657    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5355e22-6596-4c63-a492-fe8b578f04b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5355e22-6596-4c63-a492-fe8b578f04b5-istio-podinfo\") pod \"d5355e22-6596-4c63-a492-fe8b578f04b5\" (UID: \"d5355e22-6596-4c63-a492-fe8b578f04b5\") "
Oct 30 01:21:06.232853 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:06.232817    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5355e22-6596-4c63-a492-fe8b578f04b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5355e22-6596-4c63-a492-fe8b578f04b5-istio-data\") pod \"d5355e22-6596-4c63-a492-fe8b578f04b5\" (UID: \"d5355e22-6596-4c63-a492-fe8b578f04b5\") "
Oct 30 01:21:06.232918 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:06.232858    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5355e22-6596-4c63-a492-fe8b578f04b5-kube-api-access-zh26r\" (UniqueName: \"kubernetes.io/projected/d5355e22-6596-4c63-a492-fe8b578f04b5-kube-api-access-zh26r\") pod \"d5355e22-6596-4c63-a492-fe8b578f04b5\" (UID: \"d5355e22-6596-4c63-a492-fe8b578f04b5\") "
Oct 30 01:21:06.232982 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:06.232950    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5355e22-6596-4c63-a492-fe8b578f04b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5355e22-6596-4c63-a492-fe8b578f04b5-istiod-ca-cert\") pod \"d5355e22-6596-4c63-a492-fe8b578f04b5\" (UID: \"d5355e22-6596-4c63-a492-fe8b578f04b5\") "
Oct 30 01:21:06.233040 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:06.232970    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5355e22-6596-4c63-a492-fe8b578f04b5-workload-socket" pod "d5355e22-6596-4c63-a492-fe8b578f04b5" (UID: "d5355e22-6596-4c63-a492-fe8b578f04b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:06.233264 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:06.233229    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5355e22-6596-4c63-a492-fe8b578f04b5-istio-data" pod "d5355e22-6596-4c63-a492-fe8b578f04b5" (UID: "d5355e22-6596-4c63-a492-fe8b578f04b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:06.233427 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:06.233395    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5355e22-6596-4c63-a492-fe8b578f04b5-credential-socket" pod "d5355e22-6596-4c63-a492-fe8b578f04b5" (UID: "d5355e22-6596-4c63-a492-fe8b578f04b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:06.233545 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:06.233527    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5355e22-6596-4c63-a492-fe8b578f04b5-istiod-ca-cert" pod "d5355e22-6596-4c63-a492-fe8b578f04b5" (UID: "d5355e22-6596-4c63-a492-fe8b578f04b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:06.233689 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:06.233673    4393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5355e22-6596-4c63-a492-fe8b578f04b5-credential-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:06.233756 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:06.233697    4393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5355e22-6596-4c63-a492-fe8b578f04b5-workload-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:06.233756 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:06.233715    4393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5355e22-6596-4c63-a492-fe8b578f04b5-istio-data\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:06.233756 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:06.233730    4393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5355e22-6596-4c63-a492-fe8b578f04b5-istiod-ca-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:06.233943 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:06.233923    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5355e22-6596-4c63-a492-fe8b578f04b5-workload-certs" pod "d5355e22-6596-4c63-a492-fe8b578f04b5" (UID: "d5355e22-6596-4c63-a492-fe8b578f04b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:06.235714 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:06.235684    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5355e22-6596-4c63-a492-fe8b578f04b5-istio-token" pod "d5355e22-6596-4c63-a492-fe8b578f04b5" (UID: "d5355e22-6596-4c63-a492-fe8b578f04b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:06.236282 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:06.236256    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5355e22-6596-4c63-a492-fe8b578f04b5-istio-envoy" pod "d5355e22-6596-4c63-a492-fe8b578f04b5" (UID: "d5355e22-6596-4c63-a492-fe8b578f04b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:06.236515 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:06.236495    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5355e22-6596-4c63-a492-fe8b578f04b5-kube-api-access-zh26r" pod "d5355e22-6596-4c63-a492-fe8b578f04b5" (UID: "d5355e22-6596-4c63-a492-fe8b578f04b5"). InnerVolumeSpecName "kube-api-access-zh26r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:06.237449 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:06.237422    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5355e22-6596-4c63-a492-fe8b578f04b5-istio-podinfo" pod "d5355e22-6596-4c63-a492-fe8b578f04b5" (UID: "d5355e22-6596-4c63-a492-fe8b578f04b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:21:06.335521 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:06.335478    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh26r\" (UniqueName: \"kubernetes.io/projected/d5355e22-6596-4c63-a492-fe8b578f04b5-kube-api-access-zh26r\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:06.335521 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:06.335520    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5355e22-6596-4c63-a492-fe8b578f04b5-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:06.335521 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:06.335532    4393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5355e22-6596-4c63-a492-fe8b578f04b5-workload-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:06.335832 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:06.335543    4393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5355e22-6596-4c63-a492-fe8b578f04b5-istio-envoy\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:06.335832 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:06.335556    4393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5355e22-6596-4c63-a492-fe8b578f04b5-istio-podinfo\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:07.164933 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:07.164904    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"
Oct 30 01:21:07.165453 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:07.164934    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72" event={"ID":"d5355e22-6596-4c63-a492-fe8b578f04b5","Type":"ContainerDied","Data":"e347a81bbbfb67299f435cd0d4a76348b8c48e021b48a347f11925f97b4d0277"}
Oct 30 01:21:07.165453 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:07.164997    4393 scope.go:122] "RemoveContainer" containerID="12102c58a31179f367c4e9cb084c71ff9b233fd21edbcbe33ad1c0090a4d147e"
Oct 30 01:21:07.229017 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:07.228977    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"]
Oct 30 01:21:07.241581 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:07.241524    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-93079c-openshift-default-85d87fbf79-7km72"]
Oct 30 01:21:07.726228 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:07.726189    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5355e22-6596-4c63-a492-fe8b578f04b5" path="/var/lib/kubelet/pods/d5355e22-6596-4c63-a492-fe8b578f04b5/volumes"
Oct 30 01:21:21.427953 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:21.427914    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-16c30f-78c96f8ff4-dwhmx"]
Oct 30 01:21:21.428577 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:21.428154    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-16c30f-78c96f8ff4-dwhmx" podUID="de3f97cd-2787-4b28-a883-18c1f36ca325" containerName="payload-processing" containerID="cri-o://fa9fbadba700e4e5f09c71eac8c3b73ee09d5b6c84ce91baa1421263a4a194a9" gracePeriod=30
Oct 30 01:21:21.727435 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:21.726940    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-16c30f-78c96f8ff4-dwhmx"
Oct 30 01:21:21.843336 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:21.843294    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de3f97cd-2787-4b28-a883-18c1f36ca325-kube-api-access-j5pcg\" (UniqueName: \"kubernetes.io/projected/de3f97cd-2787-4b28-a883-18c1f36ca325-kube-api-access-j5pcg\") pod \"de3f97cd-2787-4b28-a883-18c1f36ca325\" (UID: \"de3f97cd-2787-4b28-a883-18c1f36ca325\") "
Oct 30 01:21:21.843844 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:21.843817    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de3f97cd-2787-4b28-a883-18c1f36ca325-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de3f97cd-2787-4b28-a883-18c1f36ca325-plugins-config-volume\") pod \"de3f97cd-2787-4b28-a883-18c1f36ca325\" (UID: \"de3f97cd-2787-4b28-a883-18c1f36ca325\") "
Oct 30 01:21:21.845986 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:21.845758    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de3f97cd-2787-4b28-a883-18c1f36ca325-plugins-config-volume" pod "de3f97cd-2787-4b28-a883-18c1f36ca325" (UID: "de3f97cd-2787-4b28-a883-18c1f36ca325"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:21.891529 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:21.891486    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de3f97cd-2787-4b28-a883-18c1f36ca325-kube-api-access-j5pcg" pod "de3f97cd-2787-4b28-a883-18c1f36ca325" (UID: "de3f97cd-2787-4b28-a883-18c1f36ca325"). InnerVolumeSpecName "kube-api-access-j5pcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:21.950348 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:21.950238    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5pcg\" (UniqueName: \"kubernetes.io/projected/de3f97cd-2787-4b28-a883-18c1f36ca325-kube-api-access-j5pcg\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:21.950348 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:21.950286    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de3f97cd-2787-4b28-a883-18c1f36ca325-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:22.309538 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:22.309432    4393 generic.go:356] "Generic (PLEG): container finished" podID="de3f97cd-2787-4b28-a883-18c1f36ca325" containerID="fa9fbadba700e4e5f09c71eac8c3b73ee09d5b6c84ce91baa1421263a4a194a9" exitCode=0
Oct 30 01:21:22.309538 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:22.309510    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-16c30f-78c96f8ff4-dwhmx"
Oct 30 01:21:22.309538 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:22.309523    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-16c30f-78c96f8ff4-dwhmx" event={"ID":"de3f97cd-2787-4b28-a883-18c1f36ca325","Type":"ContainerDied","Data":"fa9fbadba700e4e5f09c71eac8c3b73ee09d5b6c84ce91baa1421263a4a194a9"}
Oct 30 01:21:22.310008 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:22.309584    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-16c30f-78c96f8ff4-dwhmx" event={"ID":"de3f97cd-2787-4b28-a883-18c1f36ca325","Type":"ContainerDied","Data":"e7c9da53cc2f24f15cc3bcfed72b542453762fbad1a99812de18895c5ad8f3be"}
Oct 30 01:21:22.310008 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:22.309608    4393 scope.go:122] "RemoveContainer" containerID="fa9fbadba700e4e5f09c71eac8c3b73ee09d5b6c84ce91baa1421263a4a194a9"
Oct 30 01:21:22.322120 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:22.322098    4393 scope.go:122] "RemoveContainer" containerID="fa9fbadba700e4e5f09c71eac8c3b73ee09d5b6c84ce91baa1421263a4a194a9"
Oct 30 01:21:22.322448 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:21:22.322426    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa9fbadba700e4e5f09c71eac8c3b73ee09d5b6c84ce91baa1421263a4a194a9\": container with ID starting with fa9fbadba700e4e5f09c71eac8c3b73ee09d5b6c84ce91baa1421263a4a194a9 not found: ID does not exist" containerID="fa9fbadba700e4e5f09c71eac8c3b73ee09d5b6c84ce91baa1421263a4a194a9"
Oct 30 01:21:22.322529 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:22.322458    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa9fbadba700e4e5f09c71eac8c3b73ee09d5b6c84ce91baa1421263a4a194a9"} err="failed to get container status \"fa9fbadba700e4e5f09c71eac8c3b73ee09d5b6c84ce91baa1421263a4a194a9\": rpc error: code = NotFound desc = could not find container \"fa9fbadba700e4e5f09c71eac8c3b73ee09d5b6c84ce91baa1421263a4a194a9\": container with ID starting with fa9fbadba700e4e5f09c71eac8c3b73ee09d5b6c84ce91baa1421263a4a194a9 not found: ID does not exist"
Oct 30 01:21:22.361332 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:22.361295    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-16c30f-78c96f8ff4-dwhmx"]
Oct 30 01:21:22.372386 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:22.372354    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-16c30f-78c96f8ff4-dwhmx"]
Oct 30 01:21:23.745962 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:23.745919    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de3f97cd-2787-4b28-a883-18c1f36ca325" path="/var/lib/kubelet/pods/de3f97cd-2787-4b28-a883-18c1f36ca325/volumes"
Oct 30 01:21:24.342450 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:24.342408    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"]
Oct 30 01:21:24.342729 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:24.342677    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286" podUID="b31f25ea-0447-4caf-9237-d59594938f72" containerName="istio-proxy" containerID="cri-o://11cdd267f301af8315ba6fd9a2f4d97f011f70a7ad8f546fc15b2727d20ae911" gracePeriod=30
Oct 30 01:21:26.513308 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:21:26.513268    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:21:29.644484 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:29.644444    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"]
Oct 30 01:21:29.645327 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:29.645259    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz" podUID="9f083e40-d31b-4313-8459-f5ab9ac9d743" containerName="istio-proxy" containerID="cri-o://dbdef56b4ff8c0a3478dbb8dc380ffd7b4948773d125c3b9135109d3ff5bb4be" gracePeriod=30
Oct 30 01:21:29.746833 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:29.746800    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"
Oct 30 01:21:29.892226 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:29.892131    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b31f25ea-0447-4caf-9237-d59594938f72-kube-api-access-wq5n8\" (UniqueName: \"kubernetes.io/projected/b31f25ea-0447-4caf-9237-d59594938f72-kube-api-access-wq5n8\") pod \"b31f25ea-0447-4caf-9237-d59594938f72\" (UID: \"b31f25ea-0447-4caf-9237-d59594938f72\") "
Oct 30 01:21:29.892226 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:29.892179    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31f25ea-0447-4caf-9237-d59594938f72-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b31f25ea-0447-4caf-9237-d59594938f72-istio-data\") pod \"b31f25ea-0447-4caf-9237-d59594938f72\" (UID: \"b31f25ea-0447-4caf-9237-d59594938f72\") "
Oct 30 01:21:29.892472 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:29.892315    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31f25ea-0447-4caf-9237-d59594938f72-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31f25ea-0447-4caf-9237-d59594938f72-workload-socket\") pod \"b31f25ea-0447-4caf-9237-d59594938f72\" (UID: \"b31f25ea-0447-4caf-9237-d59594938f72\") "
Oct 30 01:21:29.892472 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:29.892361    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b31f25ea-0447-4caf-9237-d59594938f72-istio-token\" (UniqueName: \"kubernetes.io/projected/b31f25ea-0447-4caf-9237-d59594938f72-istio-token\") pod \"b31f25ea-0447-4caf-9237-d59594938f72\" (UID: \"b31f25ea-0447-4caf-9237-d59594938f72\") "
Oct 30 01:21:29.892472 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:29.892399    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b31f25ea-0447-4caf-9237-d59594938f72-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b31f25ea-0447-4caf-9237-d59594938f72-istiod-ca-cert\") pod \"b31f25ea-0447-4caf-9237-d59594938f72\" (UID: \"b31f25ea-0447-4caf-9237-d59594938f72\") "
Oct 30 01:21:29.892472 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:29.892435    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31f25ea-0447-4caf-9237-d59594938f72-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b31f25ea-0447-4caf-9237-d59594938f72-istio-envoy\") pod \"b31f25ea-0447-4caf-9237-d59594938f72\" (UID: \"b31f25ea-0447-4caf-9237-d59594938f72\") "
Oct 30 01:21:29.892704 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:29.892488    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31f25ea-0447-4caf-9237-d59594938f72-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31f25ea-0447-4caf-9237-d59594938f72-credential-socket\") pod \"b31f25ea-0447-4caf-9237-d59594938f72\" (UID: \"b31f25ea-0447-4caf-9237-d59594938f72\") "
Oct 30 01:21:29.892704 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:29.892526    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31f25ea-0447-4caf-9237-d59594938f72-istio-data" pod "b31f25ea-0447-4caf-9237-d59594938f72" (UID: "b31f25ea-0447-4caf-9237-d59594938f72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:29.892704 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:29.892600    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31f25ea-0447-4caf-9237-d59594938f72-workload-socket" pod "b31f25ea-0447-4caf-9237-d59594938f72" (UID: "b31f25ea-0447-4caf-9237-d59594938f72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:29.892704 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:29.892608    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31f25ea-0447-4caf-9237-d59594938f72-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b31f25ea-0447-4caf-9237-d59594938f72-workload-certs\") pod \"b31f25ea-0447-4caf-9237-d59594938f72\" (UID: \"b31f25ea-0447-4caf-9237-d59594938f72\") "
Oct 30 01:21:29.893137 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:29.892728    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b31f25ea-0447-4caf-9237-d59594938f72-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b31f25ea-0447-4caf-9237-d59594938f72-istio-podinfo\") pod \"b31f25ea-0447-4caf-9237-d59594938f72\" (UID: \"b31f25ea-0447-4caf-9237-d59594938f72\") "
Oct 30 01:21:29.893137 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:29.892794    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31f25ea-0447-4caf-9237-d59594938f72-credential-socket" pod "b31f25ea-0447-4caf-9237-d59594938f72" (UID: "b31f25ea-0447-4caf-9237-d59594938f72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:29.893137 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:29.892846    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31f25ea-0447-4caf-9237-d59594938f72-workload-certs" pod "b31f25ea-0447-4caf-9237-d59594938f72" (UID: "b31f25ea-0447-4caf-9237-d59594938f72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:29.893730 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:29.893392    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b31f25ea-0447-4caf-9237-d59594938f72-istiod-ca-cert" pod "b31f25ea-0447-4caf-9237-d59594938f72" (UID: "b31f25ea-0447-4caf-9237-d59594938f72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:29.894203 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:29.893985    4393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b31f25ea-0447-4caf-9237-d59594938f72-istiod-ca-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:29.894203 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:29.894081    4393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31f25ea-0447-4caf-9237-d59594938f72-credential-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:29.894203 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:29.894103    4393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b31f25ea-0447-4caf-9237-d59594938f72-workload-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:29.894203 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:29.894118    4393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b31f25ea-0447-4caf-9237-d59594938f72-istio-data\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:29.894203 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:29.894131    4393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31f25ea-0447-4caf-9237-d59594938f72-workload-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:29.895361 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:29.895334    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31f25ea-0447-4caf-9237-d59594938f72-istio-envoy" pod "b31f25ea-0447-4caf-9237-d59594938f72" (UID: "b31f25ea-0447-4caf-9237-d59594938f72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:29.895539 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:29.895523    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b31f25ea-0447-4caf-9237-d59594938f72-istio-token" pod "b31f25ea-0447-4caf-9237-d59594938f72" (UID: "b31f25ea-0447-4caf-9237-d59594938f72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:29.895820 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:29.895796    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b31f25ea-0447-4caf-9237-d59594938f72-kube-api-access-wq5n8" pod "b31f25ea-0447-4caf-9237-d59594938f72" (UID: "b31f25ea-0447-4caf-9237-d59594938f72"). InnerVolumeSpecName "kube-api-access-wq5n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:29.896155 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:29.896128    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b31f25ea-0447-4caf-9237-d59594938f72-istio-podinfo" pod "b31f25ea-0447-4caf-9237-d59594938f72" (UID: "b31f25ea-0447-4caf-9237-d59594938f72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:21:29.995125 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:29.995076    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b31f25ea-0447-4caf-9237-d59594938f72-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:29.995125 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:29.995116    4393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b31f25ea-0447-4caf-9237-d59594938f72-istio-envoy\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:29.995125 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:29.995130    4393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b31f25ea-0447-4caf-9237-d59594938f72-istio-podinfo\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:29.995391 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:29.995142    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq5n8\" (UniqueName: \"kubernetes.io/projected/b31f25ea-0447-4caf-9237-d59594938f72-kube-api-access-wq5n8\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:30.382641 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:30.382602    4393 generic.go:356] "Generic (PLEG): container finished" podID="b31f25ea-0447-4caf-9237-d59594938f72" containerID="11cdd267f301af8315ba6fd9a2f4d97f011f70a7ad8f546fc15b2727d20ae911" exitCode=0
Oct 30 01:21:30.382878 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:30.382669    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"
Oct 30 01:21:30.382878 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:30.382688    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286" event={"ID":"b31f25ea-0447-4caf-9237-d59594938f72","Type":"ContainerDied","Data":"11cdd267f301af8315ba6fd9a2f4d97f011f70a7ad8f546fc15b2727d20ae911"}
Oct 30 01:21:30.382878 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:30.382742    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286" event={"ID":"b31f25ea-0447-4caf-9237-d59594938f72","Type":"ContainerDied","Data":"62b676676f1e1679464503da34171bb3a705581a60d48b4dbfb1060fd90899a2"}
Oct 30 01:21:30.382878 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:30.382771    4393 scope.go:122] "RemoveContainer" containerID="11cdd267f301af8315ba6fd9a2f4d97f011f70a7ad8f546fc15b2727d20ae911"
Oct 30 01:21:30.395501 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:30.395478    4393 scope.go:122] "RemoveContainer" containerID="11cdd267f301af8315ba6fd9a2f4d97f011f70a7ad8f546fc15b2727d20ae911"
Oct 30 01:21:30.395823 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:21:30.395795    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11cdd267f301af8315ba6fd9a2f4d97f011f70a7ad8f546fc15b2727d20ae911\": container with ID starting with 11cdd267f301af8315ba6fd9a2f4d97f011f70a7ad8f546fc15b2727d20ae911 not found: ID does not exist" containerID="11cdd267f301af8315ba6fd9a2f4d97f011f70a7ad8f546fc15b2727d20ae911"
Oct 30 01:21:30.395900 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:30.395838    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11cdd267f301af8315ba6fd9a2f4d97f011f70a7ad8f546fc15b2727d20ae911"} err="failed to get container status \"11cdd267f301af8315ba6fd9a2f4d97f011f70a7ad8f546fc15b2727d20ae911\": rpc error: code = NotFound desc = could not find container \"11cdd267f301af8315ba6fd9a2f4d97f011f70a7ad8f546fc15b2727d20ae911\": container with ID starting with 11cdd267f301af8315ba6fd9a2f4d97f011f70a7ad8f546fc15b2727d20ae911 not found: ID does not exist"
Oct 30 01:21:30.434953 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:30.434915    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"]
Oct 30 01:21:30.446512 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:30.446478    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-11f521-openshift-default-96689b67c-74286"]
Oct 30 01:21:31.726547 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:31.726516    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b31f25ea-0447-4caf-9237-d59594938f72" path="/var/lib/kubelet/pods/b31f25ea-0447-4caf-9237-d59594938f72/volumes"
Oct 30 01:21:34.334006 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:34.333850    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-11f521-7d8bfbf78-q7qwq"]
Oct 30 01:21:34.348418 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:34.348385    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-11f521-6f8968b75-h6zh8"]
Oct 30 01:21:34.981079 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:34.981047    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"
Oct 30 01:21:34.993200 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:34.993162    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f083e40-d31b-4313-8459-f5ab9ac9d743-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f083e40-d31b-4313-8459-f5ab9ac9d743-istio-envoy\") pod \"9f083e40-d31b-4313-8459-f5ab9ac9d743\" (UID: \"9f083e40-d31b-4313-8459-f5ab9ac9d743\") "
Oct 30 01:21:34.993377 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:34.993229    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f083e40-d31b-4313-8459-f5ab9ac9d743-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f083e40-d31b-4313-8459-f5ab9ac9d743-workload-certs\") pod \"9f083e40-d31b-4313-8459-f5ab9ac9d743\" (UID: \"9f083e40-d31b-4313-8459-f5ab9ac9d743\") "
Oct 30 01:21:34.993377 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:34.993261    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f083e40-d31b-4313-8459-f5ab9ac9d743-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f083e40-d31b-4313-8459-f5ab9ac9d743-istiod-ca-cert\") pod \"9f083e40-d31b-4313-8459-f5ab9ac9d743\" (UID: \"9f083e40-d31b-4313-8459-f5ab9ac9d743\") "
Oct 30 01:21:34.993377 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:34.993299    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f083e40-d31b-4313-8459-f5ab9ac9d743-kube-api-access-6z47q\" (UniqueName: \"kubernetes.io/projected/9f083e40-d31b-4313-8459-f5ab9ac9d743-kube-api-access-6z47q\") pod \"9f083e40-d31b-4313-8459-f5ab9ac9d743\" (UID: \"9f083e40-d31b-4313-8459-f5ab9ac9d743\") "
Oct 30 01:21:34.993377 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:34.993330    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f083e40-d31b-4313-8459-f5ab9ac9d743-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f083e40-d31b-4313-8459-f5ab9ac9d743-credential-socket\") pod \"9f083e40-d31b-4313-8459-f5ab9ac9d743\" (UID: \"9f083e40-d31b-4313-8459-f5ab9ac9d743\") "
Oct 30 01:21:34.993712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:34.993488    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f083e40-d31b-4313-8459-f5ab9ac9d743-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f083e40-d31b-4313-8459-f5ab9ac9d743-istio-data\") pod \"9f083e40-d31b-4313-8459-f5ab9ac9d743\" (UID: \"9f083e40-d31b-4313-8459-f5ab9ac9d743\") "
Oct 30 01:21:34.993712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:34.993574    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f083e40-d31b-4313-8459-f5ab9ac9d743-workload-certs" pod "9f083e40-d31b-4313-8459-f5ab9ac9d743" (UID: "9f083e40-d31b-4313-8459-f5ab9ac9d743"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:34.993898 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:34.993744    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f083e40-d31b-4313-8459-f5ab9ac9d743-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f083e40-d31b-4313-8459-f5ab9ac9d743-workload-socket\") pod \"9f083e40-d31b-4313-8459-f5ab9ac9d743\" (UID: \"9f083e40-d31b-4313-8459-f5ab9ac9d743\") "
Oct 30 01:21:34.993898 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:34.993834    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f083e40-d31b-4313-8459-f5ab9ac9d743-istiod-ca-cert" pod "9f083e40-d31b-4313-8459-f5ab9ac9d743" (UID: "9f083e40-d31b-4313-8459-f5ab9ac9d743"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:34.993898 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:34.993864    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f083e40-d31b-4313-8459-f5ab9ac9d743-istio-token\" (UniqueName: \"kubernetes.io/projected/9f083e40-d31b-4313-8459-f5ab9ac9d743-istio-token\") pod \"9f083e40-d31b-4313-8459-f5ab9ac9d743\" (UID: \"9f083e40-d31b-4313-8459-f5ab9ac9d743\") "
Oct 30 01:21:34.994067 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:34.993956    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f083e40-d31b-4313-8459-f5ab9ac9d743-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f083e40-d31b-4313-8459-f5ab9ac9d743-istio-podinfo\") pod \"9f083e40-d31b-4313-8459-f5ab9ac9d743\" (UID: \"9f083e40-d31b-4313-8459-f5ab9ac9d743\") "
Oct 30 01:21:34.994155 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:34.993861    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f083e40-d31b-4313-8459-f5ab9ac9d743-istio-data" pod "9f083e40-d31b-4313-8459-f5ab9ac9d743" (UID: "9f083e40-d31b-4313-8459-f5ab9ac9d743"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:34.994256 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:34.994075    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f083e40-d31b-4313-8459-f5ab9ac9d743-credential-socket" pod "9f083e40-d31b-4313-8459-f5ab9ac9d743" (UID: "9f083e40-d31b-4313-8459-f5ab9ac9d743"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:34.994256 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:34.994090    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f083e40-d31b-4313-8459-f5ab9ac9d743-workload-socket" pod "9f083e40-d31b-4313-8459-f5ab9ac9d743" (UID: "9f083e40-d31b-4313-8459-f5ab9ac9d743"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:34.994643 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:34.994586    4393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f083e40-d31b-4313-8459-f5ab9ac9d743-istio-data\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:34.994643 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:34.994606    4393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f083e40-d31b-4313-8459-f5ab9ac9d743-workload-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:34.994643 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:34.994616    4393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f083e40-d31b-4313-8459-f5ab9ac9d743-workload-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:34.994643 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:34.994624    4393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f083e40-d31b-4313-8459-f5ab9ac9d743-istiod-ca-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:34.994643 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:34.994632    4393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f083e40-d31b-4313-8459-f5ab9ac9d743-credential-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:34.996608 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:34.996580    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f083e40-d31b-4313-8459-f5ab9ac9d743-istio-token" pod "9f083e40-d31b-4313-8459-f5ab9ac9d743" (UID: "9f083e40-d31b-4313-8459-f5ab9ac9d743"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:34.996812 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:34.996757    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f083e40-d31b-4313-8459-f5ab9ac9d743-kube-api-access-6z47q" pod "9f083e40-d31b-4313-8459-f5ab9ac9d743" (UID: "9f083e40-d31b-4313-8459-f5ab9ac9d743"). InnerVolumeSpecName "kube-api-access-6z47q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:34.996896 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:34.996755    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f083e40-d31b-4313-8459-f5ab9ac9d743-istio-podinfo" pod "9f083e40-d31b-4313-8459-f5ab9ac9d743" (UID: "9f083e40-d31b-4313-8459-f5ab9ac9d743"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:21:34.996896 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:34.996856    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f083e40-d31b-4313-8459-f5ab9ac9d743-istio-envoy" pod "9f083e40-d31b-4313-8459-f5ab9ac9d743" (UID: "9f083e40-d31b-4313-8459-f5ab9ac9d743"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:35.096592 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:35.096555    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f083e40-d31b-4313-8459-f5ab9ac9d743-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:35.096592 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:35.096586    4393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f083e40-d31b-4313-8459-f5ab9ac9d743-istio-podinfo\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:35.096592 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:35.096595    4393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f083e40-d31b-4313-8459-f5ab9ac9d743-istio-envoy\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:35.096592 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:35.096604    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z47q\" (UniqueName: \"kubernetes.io/projected/9f083e40-d31b-4313-8459-f5ab9ac9d743-kube-api-access-6z47q\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:35.432335 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:35.432294    4393 generic.go:356] "Generic (PLEG): container finished" podID="9f083e40-d31b-4313-8459-f5ab9ac9d743" containerID="dbdef56b4ff8c0a3478dbb8dc380ffd7b4948773d125c3b9135109d3ff5bb4be" exitCode=0
Oct 30 01:21:35.432849 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:35.432345    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"
Oct 30 01:21:35.432849 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:35.432379    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz" event={"ID":"9f083e40-d31b-4313-8459-f5ab9ac9d743","Type":"ContainerDied","Data":"dbdef56b4ff8c0a3478dbb8dc380ffd7b4948773d125c3b9135109d3ff5bb4be"}
Oct 30 01:21:35.432849 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:35.432450    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz" event={"ID":"9f083e40-d31b-4313-8459-f5ab9ac9d743","Type":"ContainerDied","Data":"1897290d1fc7439721c504fd4595596ef607edb87ea70479941ff059a3a98967"}
Oct 30 01:21:35.432849 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:35.432484    4393 scope.go:122] "RemoveContainer" containerID="dbdef56b4ff8c0a3478dbb8dc380ffd7b4948773d125c3b9135109d3ff5bb4be"
Oct 30 01:21:35.444390 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:35.444371    4393 scope.go:122] "RemoveContainer" containerID="dbdef56b4ff8c0a3478dbb8dc380ffd7b4948773d125c3b9135109d3ff5bb4be"
Oct 30 01:21:35.444682 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:21:35.444660    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbdef56b4ff8c0a3478dbb8dc380ffd7b4948773d125c3b9135109d3ff5bb4be\": container with ID starting with dbdef56b4ff8c0a3478dbb8dc380ffd7b4948773d125c3b9135109d3ff5bb4be not found: ID does not exist" containerID="dbdef56b4ff8c0a3478dbb8dc380ffd7b4948773d125c3b9135109d3ff5bb4be"
Oct 30 01:21:35.444762 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:35.444690    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbdef56b4ff8c0a3478dbb8dc380ffd7b4948773d125c3b9135109d3ff5bb4be"} err="failed to get container status \"dbdef56b4ff8c0a3478dbb8dc380ffd7b4948773d125c3b9135109d3ff5bb4be\": rpc error: code = NotFound desc = could not find container \"dbdef56b4ff8c0a3478dbb8dc380ffd7b4948773d125c3b9135109d3ff5bb4be\": container with ID starting with dbdef56b4ff8c0a3478dbb8dc380ffd7b4948773d125c3b9135109d3ff5bb4be not found: ID does not exist"
Oct 30 01:21:35.507735 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:35.507700    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"]
Oct 30 01:21:35.519126 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:35.519094    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-73435a-openshift-default-fc454477f-5kjsz"]
Oct 30 01:21:35.733427 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:35.733192    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f083e40-d31b-4313-8459-f5ab9ac9d743" path="/var/lib/kubelet/pods/9f083e40-d31b-4313-8459-f5ab9ac9d743/volumes"
Oct 30 01:21:39.185598 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:39.185558    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh"]
Oct 30 01:21:39.186132 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:39.185862    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh" podUID="6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4" containerName="maas-api" containerID="cri-o://65ed09760d48038dd2dd794ff911eb32cf15564a1a8a1393f70737ce1ebb7cd9" gracePeriod=30
Oct 30 01:21:39.334887 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:39.334833    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-11f521-7d8bfbf78-q7qwq" podUID="e1ec248b-a567-4fb4-a415-ac1235608942" containerName="payload-processing" containerID="cri-o://4783c92d9b0807d6838d68e0b0459c912aed4962ac893390169847137194cb31" gracePeriod=25
Oct 30 01:21:39.349433 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:39.349390    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-11f521-6f8968b75-h6zh8" podUID="1e044c11-fa02-43cd-af55-42b87c0b6789" containerName="payload-pre-processing" containerID="cri-o://5a6a1ca86230a4ab7704a1d3382dfcce5e519bf6d8f66d0cee9da0085acb67be" gracePeriod=25
Oct 30 01:21:39.483270 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:39.483225    4393 generic.go:356] "Generic (PLEG): container finished" podID="6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4" containerID="65ed09760d48038dd2dd794ff911eb32cf15564a1a8a1393f70737ce1ebb7cd9" exitCode=0
Oct 30 01:21:39.483520 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:39.483437    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh" event={"ID":"6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4","Type":"ContainerDied","Data":"65ed09760d48038dd2dd794ff911eb32cf15564a1a8a1393f70737ce1ebb7cd9"}
Oct 30 01:21:39.503343 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:39.503310    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh"
Oct 30 01:21:39.565421 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:39.564714    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4-openshift-service-ca\") pod \"6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4\" (UID: \"6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4\") "
Oct 30 01:21:39.565421 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:39.564809    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4-metrics-certs\") pod \"6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4\" (UID: \"6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4\") "
Oct 30 01:21:39.565421 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:39.564844    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4-kube-api-access-jncfh\" (UniqueName: \"kubernetes.io/projected/6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4-kube-api-access-jncfh\") pod \"6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4\" (UID: \"6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4\") "
Oct 30 01:21:39.565421 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:39.564973    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4-maas-api-tls\") pod \"6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4\" (UID: \"6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4\") "
Oct 30 01:21:39.566749 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:39.566721    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4-openshift-service-ca" pod "6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4" (UID: "6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:39.571199 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:39.571166    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4-metrics-certs" pod "6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4" (UID: "6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:21:39.571339 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:39.571276    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4-maas-api-tls" pod "6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4" (UID: "6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:21:39.573018 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:39.572993    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4-kube-api-access-jncfh" pod "6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4" (UID: "6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4"). InnerVolumeSpecName "kube-api-access-jncfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:39.638179 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:39.638152    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-11f521-6f8968b75-h6zh8"
Oct 30 01:21:39.667715 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:39.667666    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:39.667715 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:39.667710    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:39.667715 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:39.667728    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:39.667715 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:39.667743    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jncfh\" (UniqueName: \"kubernetes.io/projected/6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4-kube-api-access-jncfh\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:39.769570 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:39.769528    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e044c11-fa02-43cd-af55-42b87c0b6789-kube-api-access-k5rkd\" (UniqueName: \"kubernetes.io/projected/1e044c11-fa02-43cd-af55-42b87c0b6789-kube-api-access-k5rkd\") pod \"1e044c11-fa02-43cd-af55-42b87c0b6789\" (UID: \"1e044c11-fa02-43cd-af55-42b87c0b6789\") "
Oct 30 01:21:39.769847 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:39.769654    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e044c11-fa02-43cd-af55-42b87c0b6789-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e044c11-fa02-43cd-af55-42b87c0b6789-plugins-config-volume\") pod \"1e044c11-fa02-43cd-af55-42b87c0b6789\" (UID: \"1e044c11-fa02-43cd-af55-42b87c0b6789\") "
Oct 30 01:21:39.770549 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:39.770509    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e044c11-fa02-43cd-af55-42b87c0b6789-plugins-config-volume" pod "1e044c11-fa02-43cd-af55-42b87c0b6789" (UID: "1e044c11-fa02-43cd-af55-42b87c0b6789"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:39.772583 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:39.771309    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e044c11-fa02-43cd-af55-42b87c0b6789-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:39.776383 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:39.776343    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e044c11-fa02-43cd-af55-42b87c0b6789-kube-api-access-k5rkd" pod "1e044c11-fa02-43cd-af55-42b87c0b6789" (UID: "1e044c11-fa02-43cd-af55-42b87c0b6789"). InnerVolumeSpecName "kube-api-access-k5rkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:39.778880 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:39.778858    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-11f521-7d8bfbf78-q7qwq"
Oct 30 01:21:39.872745 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:39.872712    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1ec248b-a567-4fb4-a415-ac1235608942-kube-api-access-88s9p\" (UniqueName: \"kubernetes.io/projected/e1ec248b-a567-4fb4-a415-ac1235608942-kube-api-access-88s9p\") pod \"e1ec248b-a567-4fb4-a415-ac1235608942\" (UID: \"e1ec248b-a567-4fb4-a415-ac1235608942\") "
Oct 30 01:21:39.872976 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:39.872961    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1ec248b-a567-4fb4-a415-ac1235608942-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1ec248b-a567-4fb4-a415-ac1235608942-plugins-config-volume\") pod \"e1ec248b-a567-4fb4-a415-ac1235608942\" (UID: \"e1ec248b-a567-4fb4-a415-ac1235608942\") "
Oct 30 01:21:39.873624 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:39.873394    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1ec248b-a567-4fb4-a415-ac1235608942-plugins-config-volume" pod "e1ec248b-a567-4fb4-a415-ac1235608942" (UID: "e1ec248b-a567-4fb4-a415-ac1235608942"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:39.873624 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:39.873424    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5rkd\" (UniqueName: \"kubernetes.io/projected/1e044c11-fa02-43cd-af55-42b87c0b6789-kube-api-access-k5rkd\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:39.875024 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:39.874993    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1ec248b-a567-4fb4-a415-ac1235608942-kube-api-access-88s9p" pod "e1ec248b-a567-4fb4-a415-ac1235608942" (UID: "e1ec248b-a567-4fb4-a415-ac1235608942"). InnerVolumeSpecName "kube-api-access-88s9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:39.982170 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:39.982045    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1ec248b-a567-4fb4-a415-ac1235608942-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:39.982170 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:39.982084    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88s9p\" (UniqueName: \"kubernetes.io/projected/e1ec248b-a567-4fb4-a415-ac1235608942-kube-api-access-88s9p\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:40.495679 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:40.495650    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh"
Oct 30 01:21:40.496431 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:40.495665    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh" event={"ID":"6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4","Type":"ContainerDied","Data":"358b354d57aa8955adadc7ccc04f469bd86c5d01aa54440b9e05ee39b6863b97"}
Oct 30 01:21:40.496524 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:40.496495    4393 scope.go:122] "RemoveContainer" containerID="65ed09760d48038dd2dd794ff911eb32cf15564a1a8a1393f70737ce1ebb7cd9"
Oct 30 01:21:40.502091 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:40.499738    4393 generic.go:356] "Generic (PLEG): container finished" podID="1e044c11-fa02-43cd-af55-42b87c0b6789" containerID="5a6a1ca86230a4ab7704a1d3382dfcce5e519bf6d8f66d0cee9da0085acb67be" exitCode=0
Oct 30 01:21:40.502091 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:40.500327    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-11f521-6f8968b75-h6zh8" event={"ID":"1e044c11-fa02-43cd-af55-42b87c0b6789","Type":"ContainerDied","Data":"5a6a1ca86230a4ab7704a1d3382dfcce5e519bf6d8f66d0cee9da0085acb67be"}
Oct 30 01:21:40.502091 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:40.500383    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-11f521-6f8968b75-h6zh8" event={"ID":"1e044c11-fa02-43cd-af55-42b87c0b6789","Type":"ContainerDied","Data":"696379b0d6f5b89ba993c766a2baa08f110b79320cc658bf7089a77326cb3b6d"}
Oct 30 01:21:40.502091 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:40.500465    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-11f521-6f8968b75-h6zh8"
Oct 30 01:21:40.518210 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:40.517810    4393 scope.go:122] "RemoveContainer" containerID="5a6a1ca86230a4ab7704a1d3382dfcce5e519bf6d8f66d0cee9da0085acb67be"
Oct 30 01:21:40.520537 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:40.520511    4393 generic.go:356] "Generic (PLEG): container finished" podID="e1ec248b-a567-4fb4-a415-ac1235608942" containerID="4783c92d9b0807d6838d68e0b0459c912aed4962ac893390169847137194cb31" exitCode=0
Oct 30 01:21:40.520855 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:40.520587    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-11f521-7d8bfbf78-q7qwq"
Oct 30 01:21:40.521029 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:40.520626    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-11f521-7d8bfbf78-q7qwq" event={"ID":"e1ec248b-a567-4fb4-a415-ac1235608942","Type":"ContainerDied","Data":"4783c92d9b0807d6838d68e0b0459c912aed4962ac893390169847137194cb31"}
Oct 30 01:21:40.521197 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:40.521180    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-11f521-7d8bfbf78-q7qwq" event={"ID":"e1ec248b-a567-4fb4-a415-ac1235608942","Type":"ContainerDied","Data":"a95d868386915aaf291dfad18f24ab589764c602dcdbf3e2d927ae07a400dfef"}
Oct 30 01:21:40.535007 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:40.534990    4393 scope.go:122] "RemoveContainer" containerID="5a6a1ca86230a4ab7704a1d3382dfcce5e519bf6d8f66d0cee9da0085acb67be"
Oct 30 01:21:40.535403 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:21:40.535376    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a6a1ca86230a4ab7704a1d3382dfcce5e519bf6d8f66d0cee9da0085acb67be\": container with ID starting with 5a6a1ca86230a4ab7704a1d3382dfcce5e519bf6d8f66d0cee9da0085acb67be not found: ID does not exist" containerID="5a6a1ca86230a4ab7704a1d3382dfcce5e519bf6d8f66d0cee9da0085acb67be"
Oct 30 01:21:40.535525 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:40.535409    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a6a1ca86230a4ab7704a1d3382dfcce5e519bf6d8f66d0cee9da0085acb67be"} err="failed to get container status \"5a6a1ca86230a4ab7704a1d3382dfcce5e519bf6d8f66d0cee9da0085acb67be\": rpc error: code = NotFound desc = could not find container \"5a6a1ca86230a4ab7704a1d3382dfcce5e519bf6d8f66d0cee9da0085acb67be\": container with ID starting with 5a6a1ca86230a4ab7704a1d3382dfcce5e519bf6d8f66d0cee9da0085acb67be not found: ID does not exist"
Oct 30 01:21:40.535525 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:40.535426    4393 scope.go:122] "RemoveContainer" containerID="4783c92d9b0807d6838d68e0b0459c912aed4962ac893390169847137194cb31"
Oct 30 01:21:40.546547 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:40.546478    4393 scope.go:122] "RemoveContainer" containerID="4783c92d9b0807d6838d68e0b0459c912aed4962ac893390169847137194cb31"
Oct 30 01:21:40.546833 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:21:40.546813    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4783c92d9b0807d6838d68e0b0459c912aed4962ac893390169847137194cb31\": container with ID starting with 4783c92d9b0807d6838d68e0b0459c912aed4962ac893390169847137194cb31 not found: ID does not exist" containerID="4783c92d9b0807d6838d68e0b0459c912aed4962ac893390169847137194cb31"
Oct 30 01:21:40.546892 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:40.546842    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4783c92d9b0807d6838d68e0b0459c912aed4962ac893390169847137194cb31"} err="failed to get container status \"4783c92d9b0807d6838d68e0b0459c912aed4962ac893390169847137194cb31\": rpc error: code = NotFound desc = could not find container \"4783c92d9b0807d6838d68e0b0459c912aed4962ac893390169847137194cb31\": container with ID starting with 4783c92d9b0807d6838d68e0b0459c912aed4962ac893390169847137194cb31 not found: ID does not exist"
Oct 30 01:21:40.556043 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:40.556020    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh"]
Oct 30 01:21:40.570812 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:40.570768    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-a5e9ab-6f9df9bc46-r6dnh"]
Oct 30 01:21:40.607611 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:40.607577    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-11f521-6f8968b75-h6zh8"]
Oct 30 01:21:40.624652 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:40.624617    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-11f521-6f8968b75-h6zh8"]
Oct 30 01:21:40.655912 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:40.655860    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-11f521-7d8bfbf78-q7qwq"]
Oct 30 01:21:40.659717 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:40.659686    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-11f521-7d8bfbf78-q7qwq"]
Oct 30 01:21:41.726085 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:41.726049    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e044c11-fa02-43cd-af55-42b87c0b6789" path="/var/lib/kubelet/pods/1e044c11-fa02-43cd-af55-42b87c0b6789/volumes"
Oct 30 01:21:41.726659 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:41.726634    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4" path="/var/lib/kubelet/pods/6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4/volumes"
Oct 30 01:21:41.727100 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:41.727085    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1ec248b-a567-4fb4-a415-ac1235608942" path="/var/lib/kubelet/pods/e1ec248b-a567-4fb4-a415-ac1235608942/volumes"
Oct 30 01:21:45.086526 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:45.086483    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"]
Oct 30 01:21:45.087036 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:45.086753    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp" podUID="1e34134b-9305-4376-b55b-c08c72cfbedd" containerName="istio-proxy" containerID="cri-o://885834e85744070f20a70adb7d0bf6a74fd431d662c798875de1b6be267e6bc3" gracePeriod=30
Oct 30 01:21:50.401755 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.401724    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"
Oct 30 01:21:50.470413 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.469001    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e34134b-9305-4376-b55b-c08c72cfbedd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e34134b-9305-4376-b55b-c08c72cfbedd-istiod-ca-cert\") pod \"1e34134b-9305-4376-b55b-c08c72cfbedd\" (UID: \"1e34134b-9305-4376-b55b-c08c72cfbedd\") "
Oct 30 01:21:50.470413 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.469105    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e34134b-9305-4376-b55b-c08c72cfbedd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e34134b-9305-4376-b55b-c08c72cfbedd-istio-data\") pod \"1e34134b-9305-4376-b55b-c08c72cfbedd\" (UID: \"1e34134b-9305-4376-b55b-c08c72cfbedd\") "
Oct 30 01:21:50.470413 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.469145    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e34134b-9305-4376-b55b-c08c72cfbedd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e34134b-9305-4376-b55b-c08c72cfbedd-credential-socket\") pod \"1e34134b-9305-4376-b55b-c08c72cfbedd\" (UID: \"1e34134b-9305-4376-b55b-c08c72cfbedd\") "
Oct 30 01:21:50.470413 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.469206    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e34134b-9305-4376-b55b-c08c72cfbedd-kube-api-access-r42zf\" (UniqueName: \"kubernetes.io/projected/1e34134b-9305-4376-b55b-c08c72cfbedd-kube-api-access-r42zf\") pod \"1e34134b-9305-4376-b55b-c08c72cfbedd\" (UID: \"1e34134b-9305-4376-b55b-c08c72cfbedd\") "
Oct 30 01:21:50.470413 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.469269    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e34134b-9305-4376-b55b-c08c72cfbedd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e34134b-9305-4376-b55b-c08c72cfbedd-istio-envoy\") pod \"1e34134b-9305-4376-b55b-c08c72cfbedd\" (UID: \"1e34134b-9305-4376-b55b-c08c72cfbedd\") "
Oct 30 01:21:50.470413 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.469340    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e34134b-9305-4376-b55b-c08c72cfbedd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e34134b-9305-4376-b55b-c08c72cfbedd-workload-socket\") pod \"1e34134b-9305-4376-b55b-c08c72cfbedd\" (UID: \"1e34134b-9305-4376-b55b-c08c72cfbedd\") "
Oct 30 01:21:50.470413 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.469400    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e34134b-9305-4376-b55b-c08c72cfbedd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e34134b-9305-4376-b55b-c08c72cfbedd-workload-certs\") pod \"1e34134b-9305-4376-b55b-c08c72cfbedd\" (UID: \"1e34134b-9305-4376-b55b-c08c72cfbedd\") "
Oct 30 01:21:50.470413 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.469455    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e34134b-9305-4376-b55b-c08c72cfbedd-istio-token\" (UniqueName: \"kubernetes.io/projected/1e34134b-9305-4376-b55b-c08c72cfbedd-istio-token\") pod \"1e34134b-9305-4376-b55b-c08c72cfbedd\" (UID: \"1e34134b-9305-4376-b55b-c08c72cfbedd\") "
Oct 30 01:21:50.470413 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.469481    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e34134b-9305-4376-b55b-c08c72cfbedd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e34134b-9305-4376-b55b-c08c72cfbedd-istio-podinfo\") pod \"1e34134b-9305-4376-b55b-c08c72cfbedd\" (UID: \"1e34134b-9305-4376-b55b-c08c72cfbedd\") "
Oct 30 01:21:50.471034 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.470730    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e34134b-9305-4376-b55b-c08c72cfbedd-istio-data" pod "1e34134b-9305-4376-b55b-c08c72cfbedd" (UID: "1e34134b-9305-4376-b55b-c08c72cfbedd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:50.471104 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.471068    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e34134b-9305-4376-b55b-c08c72cfbedd-istiod-ca-cert" pod "1e34134b-9305-4376-b55b-c08c72cfbedd" (UID: "1e34134b-9305-4376-b55b-c08c72cfbedd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:50.471859 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.471830    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e34134b-9305-4376-b55b-c08c72cfbedd-workload-socket" pod "1e34134b-9305-4376-b55b-c08c72cfbedd" (UID: "1e34134b-9305-4376-b55b-c08c72cfbedd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:50.474113 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.473096    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e34134b-9305-4376-b55b-c08c72cfbedd-workload-certs" pod "1e34134b-9305-4376-b55b-c08c72cfbedd" (UID: "1e34134b-9305-4376-b55b-c08c72cfbedd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:50.474113 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.473299    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e34134b-9305-4376-b55b-c08c72cfbedd-credential-socket" pod "1e34134b-9305-4376-b55b-c08c72cfbedd" (UID: "1e34134b-9305-4376-b55b-c08c72cfbedd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:50.475185 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.475160    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e34134b-9305-4376-b55b-c08c72cfbedd-kube-api-access-r42zf" pod "1e34134b-9305-4376-b55b-c08c72cfbedd" (UID: "1e34134b-9305-4376-b55b-c08c72cfbedd"). InnerVolumeSpecName "kube-api-access-r42zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:50.475722 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.475693    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e34134b-9305-4376-b55b-c08c72cfbedd-istio-podinfo" pod "1e34134b-9305-4376-b55b-c08c72cfbedd" (UID: "1e34134b-9305-4376-b55b-c08c72cfbedd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:21:50.476012 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.475989    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e34134b-9305-4376-b55b-c08c72cfbedd-istio-envoy" pod "1e34134b-9305-4376-b55b-c08c72cfbedd" (UID: "1e34134b-9305-4376-b55b-c08c72cfbedd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:21:50.476664 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.476529    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e34134b-9305-4376-b55b-c08c72cfbedd-istio-token" pod "1e34134b-9305-4376-b55b-c08c72cfbedd" (UID: "1e34134b-9305-4376-b55b-c08c72cfbedd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:50.571669 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.571631    4393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e34134b-9305-4376-b55b-c08c72cfbedd-istio-data\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:50.571669 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.571668    4393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e34134b-9305-4376-b55b-c08c72cfbedd-credential-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:50.571669 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.571678    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r42zf\" (UniqueName: \"kubernetes.io/projected/1e34134b-9305-4376-b55b-c08c72cfbedd-kube-api-access-r42zf\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:50.572282 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.571688    4393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e34134b-9305-4376-b55b-c08c72cfbedd-istio-envoy\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:50.572282 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.571699    4393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e34134b-9305-4376-b55b-c08c72cfbedd-workload-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:50.572282 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.571710    4393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e34134b-9305-4376-b55b-c08c72cfbedd-workload-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:50.572282 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.571719    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e34134b-9305-4376-b55b-c08c72cfbedd-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:50.572282 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.571726    4393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e34134b-9305-4376-b55b-c08c72cfbedd-istio-podinfo\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:50.572282 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.571734    4393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e34134b-9305-4376-b55b-c08c72cfbedd-istiod-ca-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:50.616657 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.616616    4393 generic.go:356] "Generic (PLEG): container finished" podID="1e34134b-9305-4376-b55b-c08c72cfbedd" containerID="885834e85744070f20a70adb7d0bf6a74fd431d662c798875de1b6be267e6bc3" exitCode=0
Oct 30 01:21:50.617165 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.616677    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"
Oct 30 01:21:50.617165 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.616700    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp" event={"ID":"1e34134b-9305-4376-b55b-c08c72cfbedd","Type":"ContainerDied","Data":"885834e85744070f20a70adb7d0bf6a74fd431d662c798875de1b6be267e6bc3"}
Oct 30 01:21:50.617165 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.616768    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp" event={"ID":"1e34134b-9305-4376-b55b-c08c72cfbedd","Type":"ContainerDied","Data":"f309f35955db877fca413297bd6fc31b6d6d056923dd48ff0c9e2e92f5ed8c5f"}
Oct 30 01:21:50.617165 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.616874    4393 scope.go:122] "RemoveContainer" containerID="885834e85744070f20a70adb7d0bf6a74fd431d662c798875de1b6be267e6bc3"
Oct 30 01:21:50.629764 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.629736    4393 scope.go:122] "RemoveContainer" containerID="885834e85744070f20a70adb7d0bf6a74fd431d662c798875de1b6be267e6bc3"
Oct 30 01:21:50.630126 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:21:50.630102    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"885834e85744070f20a70adb7d0bf6a74fd431d662c798875de1b6be267e6bc3\": container with ID starting with 885834e85744070f20a70adb7d0bf6a74fd431d662c798875de1b6be267e6bc3 not found: ID does not exist" containerID="885834e85744070f20a70adb7d0bf6a74fd431d662c798875de1b6be267e6bc3"
Oct 30 01:21:50.630180 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.630139    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"885834e85744070f20a70adb7d0bf6a74fd431d662c798875de1b6be267e6bc3"} err="failed to get container status \"885834e85744070f20a70adb7d0bf6a74fd431d662c798875de1b6be267e6bc3\": rpc error: code = NotFound desc = could not find container \"885834e85744070f20a70adb7d0bf6a74fd431d662c798875de1b6be267e6bc3\": container with ID starting with 885834e85744070f20a70adb7d0bf6a74fd431d662c798875de1b6be267e6bc3 not found: ID does not exist"
Oct 30 01:21:50.669458 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.669423    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"]
Oct 30 01:21:50.673352 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:50.673317    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-009e78-openshift-default-5674f58c7-znnhp"]
Oct 30 01:21:51.725864 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:51.725830    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e34134b-9305-4376-b55b-c08c72cfbedd" path="/var/lib/kubelet/pods/1e34134b-9305-4376-b55b-c08c72cfbedd/volumes"
Oct 30 01:21:51.797682 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:51.797638    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52"]
Oct 30 01:21:51.798019 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:51.797928    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52" podUID="a88ea59f-24dc-42df-8bd5-ffffdb5f36f2" containerName="maas-api" containerID="cri-o://38d7860401974c4370a1853fd2c2a72b9470a34d61a57e7097a771479275b2bc" gracePeriod=30
Oct 30 01:21:52.120372 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:52.120347    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52"
Oct 30 01:21:52.193264 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:52.193229    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a88ea59f-24dc-42df-8bd5-ffffdb5f36f2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a88ea59f-24dc-42df-8bd5-ffffdb5f36f2-openshift-service-ca\") pod \"a88ea59f-24dc-42df-8bd5-ffffdb5f36f2\" (UID: \"a88ea59f-24dc-42df-8bd5-ffffdb5f36f2\") "
Oct 30 01:21:52.193473 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:52.193284    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a88ea59f-24dc-42df-8bd5-ffffdb5f36f2-kube-api-access-hmfvd\" (UniqueName: \"kubernetes.io/projected/a88ea59f-24dc-42df-8bd5-ffffdb5f36f2-kube-api-access-hmfvd\") pod \"a88ea59f-24dc-42df-8bd5-ffffdb5f36f2\" (UID: \"a88ea59f-24dc-42df-8bd5-ffffdb5f36f2\") "
Oct 30 01:21:52.193473 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:52.193433    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a88ea59f-24dc-42df-8bd5-ffffdb5f36f2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a88ea59f-24dc-42df-8bd5-ffffdb5f36f2-metrics-certs\") pod \"a88ea59f-24dc-42df-8bd5-ffffdb5f36f2\" (UID: \"a88ea59f-24dc-42df-8bd5-ffffdb5f36f2\") "
Oct 30 01:21:52.193603 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:52.193484    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a88ea59f-24dc-42df-8bd5-ffffdb5f36f2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a88ea59f-24dc-42df-8bd5-ffffdb5f36f2-maas-api-tls\") pod \"a88ea59f-24dc-42df-8bd5-ffffdb5f36f2\" (UID: \"a88ea59f-24dc-42df-8bd5-ffffdb5f36f2\") "
Oct 30 01:21:52.193713 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:52.193685    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a88ea59f-24dc-42df-8bd5-ffffdb5f36f2-openshift-service-ca" pod "a88ea59f-24dc-42df-8bd5-ffffdb5f36f2" (UID: "a88ea59f-24dc-42df-8bd5-ffffdb5f36f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:21:52.194172 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:52.194135    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a88ea59f-24dc-42df-8bd5-ffffdb5f36f2-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:52.195821 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:52.195762    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a88ea59f-24dc-42df-8bd5-ffffdb5f36f2-kube-api-access-hmfvd" pod "a88ea59f-24dc-42df-8bd5-ffffdb5f36f2" (UID: "a88ea59f-24dc-42df-8bd5-ffffdb5f36f2"). InnerVolumeSpecName "kube-api-access-hmfvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:21:52.196302 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:52.196274    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a88ea59f-24dc-42df-8bd5-ffffdb5f36f2-maas-api-tls" pod "a88ea59f-24dc-42df-8bd5-ffffdb5f36f2" (UID: "a88ea59f-24dc-42df-8bd5-ffffdb5f36f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:21:52.196379 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:52.196363    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a88ea59f-24dc-42df-8bd5-ffffdb5f36f2-metrics-certs" pod "a88ea59f-24dc-42df-8bd5-ffffdb5f36f2" (UID: "a88ea59f-24dc-42df-8bd5-ffffdb5f36f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:21:52.314898 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:52.297094    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a88ea59f-24dc-42df-8bd5-ffffdb5f36f2-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:52.314898 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:52.297153    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a88ea59f-24dc-42df-8bd5-ffffdb5f36f2-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:52.314898 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:52.299855    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmfvd\" (UniqueName: \"kubernetes.io/projected/a88ea59f-24dc-42df-8bd5-ffffdb5f36f2-kube-api-access-hmfvd\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:21:52.633833 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:52.633751    4393 generic.go:356] "Generic (PLEG): container finished" podID="a88ea59f-24dc-42df-8bd5-ffffdb5f36f2" containerID="38d7860401974c4370a1853fd2c2a72b9470a34d61a57e7097a771479275b2bc" exitCode=0
Oct 30 01:21:52.633833 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:52.633809    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52" event={"ID":"a88ea59f-24dc-42df-8bd5-ffffdb5f36f2","Type":"ContainerDied","Data":"38d7860401974c4370a1853fd2c2a72b9470a34d61a57e7097a771479275b2bc"}
Oct 30 01:21:52.633833 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:52.633833    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52"
Oct 30 01:21:52.634126 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:52.633870    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52" event={"ID":"a88ea59f-24dc-42df-8bd5-ffffdb5f36f2","Type":"ContainerDied","Data":"30227892e2becfbbf6ea66bdd84b4a77f8afaf40dd8e6959e52eae6efd176ae4"}
Oct 30 01:21:52.634126 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:52.633900    4393 scope.go:122] "RemoveContainer" containerID="38d7860401974c4370a1853fd2c2a72b9470a34d61a57e7097a771479275b2bc"
Oct 30 01:21:52.645216 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:52.645192    4393 scope.go:122] "RemoveContainer" containerID="38d7860401974c4370a1853fd2c2a72b9470a34d61a57e7097a771479275b2bc"
Oct 30 01:21:52.645524 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:21:52.645493    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38d7860401974c4370a1853fd2c2a72b9470a34d61a57e7097a771479275b2bc\": container with ID starting with 38d7860401974c4370a1853fd2c2a72b9470a34d61a57e7097a771479275b2bc not found: ID does not exist" containerID="38d7860401974c4370a1853fd2c2a72b9470a34d61a57e7097a771479275b2bc"
Oct 30 01:21:52.645639 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:52.645523    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38d7860401974c4370a1853fd2c2a72b9470a34d61a57e7097a771479275b2bc"} err="failed to get container status \"38d7860401974c4370a1853fd2c2a72b9470a34d61a57e7097a771479275b2bc\": rpc error: code = NotFound desc = could not find container \"38d7860401974c4370a1853fd2c2a72b9470a34d61a57e7097a771479275b2bc\": container with ID starting with 38d7860401974c4370a1853fd2c2a72b9470a34d61a57e7097a771479275b2bc not found: ID does not exist"
Oct 30 01:21:52.707393 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:52.707338    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52"]
Oct 30 01:21:52.712013 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:52.711979    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-11f521-5d858988b4-2xj52"]
Oct 30 01:21:53.725552 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:53.725520    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a88ea59f-24dc-42df-8bd5-ffffdb5f36f2" path="/var/lib/kubelet/pods/a88ea59f-24dc-42df-8bd5-ffffdb5f36f2/volumes"
Oct 30 01:21:56.512573 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:21:56.512528    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:21:58.119623 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:58.119584    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-6f4fba-86877b49f8-n59lf"]
Oct 30 01:21:58.149824 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:21:58.149769    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-6f4fba-5c479c7dd7-9sxbh"]
Oct 30 01:22:03.120661 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:03.120539    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-6f4fba-86877b49f8-n59lf" podUID="2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd" containerName="payload-processing" containerID="cri-o://534690997e7276a0920e37caa6a65c204cc478d8b80679c92581c09f1602647a" gracePeriod=25
Oct 30 01:22:03.150944 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:03.150902    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6f4fba-5c479c7dd7-9sxbh" podUID="44688868-ab51-4384-9887-b1d61810eca3" containerName="payload-pre-processing" containerID="cri-o://87814af2e15bf8ebe57f4f8606f4f7fa5a87f4e1b13698355bae99dd53f47e09" gracePeriod=25
Oct 30 01:22:03.449495 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:03.449462    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-6f4fba-86877b49f8-n59lf"
Oct 30 01:22:03.480823 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:03.480763    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6f4fba-5c479c7dd7-9sxbh"
Oct 30 01:22:03.573491 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:03.573451    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd-kube-api-access-7czbv\" (UniqueName: \"kubernetes.io/projected/2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd-kube-api-access-7czbv\") pod \"2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd\" (UID: \"2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd\") "
Oct 30 01:22:03.573685 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:03.573573    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44688868-ab51-4384-9887-b1d61810eca3-kube-api-access-fnjnt\" (UniqueName: \"kubernetes.io/projected/44688868-ab51-4384-9887-b1d61810eca3-kube-api-access-fnjnt\") pod \"44688868-ab51-4384-9887-b1d61810eca3\" (UID: \"44688868-ab51-4384-9887-b1d61810eca3\") "
Oct 30 01:22:03.573729 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:03.573696    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd-plugins-config-volume\") pod \"2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd\" (UID: \"2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd\") "
Oct 30 01:22:03.573791 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:03.573754    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44688868-ab51-4384-9887-b1d61810eca3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44688868-ab51-4384-9887-b1d61810eca3-plugins-config-volume\") pod \"44688868-ab51-4384-9887-b1d61810eca3\" (UID: \"44688868-ab51-4384-9887-b1d61810eca3\") "
Oct 30 01:22:03.574168 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:03.574136    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44688868-ab51-4384-9887-b1d61810eca3-plugins-config-volume" pod "44688868-ab51-4384-9887-b1d61810eca3" (UID: "44688868-ab51-4384-9887-b1d61810eca3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:03.574168 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:03.574137    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd-plugins-config-volume" pod "2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd" (UID: "2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:03.574513 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:03.574492    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:03.574558 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:03.574517    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44688868-ab51-4384-9887-b1d61810eca3-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:03.576084 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:03.576056    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44688868-ab51-4384-9887-b1d61810eca3-kube-api-access-fnjnt" pod "44688868-ab51-4384-9887-b1d61810eca3" (UID: "44688868-ab51-4384-9887-b1d61810eca3"). InnerVolumeSpecName "kube-api-access-fnjnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:03.576183 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:03.576165    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd-kube-api-access-7czbv" pod "2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd" (UID: "2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd"). InnerVolumeSpecName "kube-api-access-7czbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:03.676300 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:03.676206    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7czbv\" (UniqueName: \"kubernetes.io/projected/2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd-kube-api-access-7czbv\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:03.676300 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:03.676243    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnjnt\" (UniqueName: \"kubernetes.io/projected/44688868-ab51-4384-9887-b1d61810eca3-kube-api-access-fnjnt\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:03.715845 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:03.715804    4393 generic.go:356] "Generic (PLEG): container finished" podID="44688868-ab51-4384-9887-b1d61810eca3" containerID="87814af2e15bf8ebe57f4f8606f4f7fa5a87f4e1b13698355bae99dd53f47e09" exitCode=0
Oct 30 01:22:03.716047 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:03.715860    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6f4fba-5c479c7dd7-9sxbh"
Oct 30 01:22:03.716047 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:03.715878    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6f4fba-5c479c7dd7-9sxbh" event={"ID":"44688868-ab51-4384-9887-b1d61810eca3","Type":"ContainerDied","Data":"87814af2e15bf8ebe57f4f8606f4f7fa5a87f4e1b13698355bae99dd53f47e09"}
Oct 30 01:22:03.716047 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:03.715951    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-6f4fba-5c479c7dd7-9sxbh" event={"ID":"44688868-ab51-4384-9887-b1d61810eca3","Type":"ContainerDied","Data":"8cc9e2bc7ea42b3cbbcbb8929226ca0e3baee4b3762917175931e19c0c68a7bb"}
Oct 30 01:22:03.716047 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:03.715974    4393 scope.go:122] "RemoveContainer" containerID="87814af2e15bf8ebe57f4f8606f4f7fa5a87f4e1b13698355bae99dd53f47e09"
Oct 30 01:22:03.717848 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:03.717816    4393 generic.go:356] "Generic (PLEG): container finished" podID="2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd" containerID="534690997e7276a0920e37caa6a65c204cc478d8b80679c92581c09f1602647a" exitCode=0
Oct 30 01:22:03.718020 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:03.717974    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-6f4fba-86877b49f8-n59lf"
Oct 30 01:22:03.729314 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:03.729107    4393 scope.go:122] "RemoveContainer" containerID="87814af2e15bf8ebe57f4f8606f4f7fa5a87f4e1b13698355bae99dd53f47e09"
Oct 30 01:22:03.729314 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:03.729259    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-6f4fba-86877b49f8-n59lf" event={"ID":"2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd","Type":"ContainerDied","Data":"534690997e7276a0920e37caa6a65c204cc478d8b80679c92581c09f1602647a"}
Oct 30 01:22:03.729314 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:03.729312    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-6f4fba-86877b49f8-n59lf" event={"ID":"2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd","Type":"ContainerDied","Data":"9b392dfd47b6ee4a8c7b53ad524baca294d22e44c1acfabbcf94bf6cff183833"}
Oct 30 01:22:03.729600 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:22:03.729465    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87814af2e15bf8ebe57f4f8606f4f7fa5a87f4e1b13698355bae99dd53f47e09\": container with ID starting with 87814af2e15bf8ebe57f4f8606f4f7fa5a87f4e1b13698355bae99dd53f47e09 not found: ID does not exist" containerID="87814af2e15bf8ebe57f4f8606f4f7fa5a87f4e1b13698355bae99dd53f47e09"
Oct 30 01:22:03.729600 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:03.729499    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87814af2e15bf8ebe57f4f8606f4f7fa5a87f4e1b13698355bae99dd53f47e09"} err="failed to get container status \"87814af2e15bf8ebe57f4f8606f4f7fa5a87f4e1b13698355bae99dd53f47e09\": rpc error: code = NotFound desc = could not find container \"87814af2e15bf8ebe57f4f8606f4f7fa5a87f4e1b13698355bae99dd53f47e09\": container with ID starting with 87814af2e15bf8ebe57f4f8606f4f7fa5a87f4e1b13698355bae99dd53f47e09 not found: ID does not exist"
Oct 30 01:22:03.729600 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:03.729525    4393 scope.go:122] "RemoveContainer" containerID="534690997e7276a0920e37caa6a65c204cc478d8b80679c92581c09f1602647a"
Oct 30 01:22:03.741728 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:03.741706    4393 scope.go:122] "RemoveContainer" containerID="534690997e7276a0920e37caa6a65c204cc478d8b80679c92581c09f1602647a"
Oct 30 01:22:03.742110 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:22:03.742088    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"534690997e7276a0920e37caa6a65c204cc478d8b80679c92581c09f1602647a\": container with ID starting with 534690997e7276a0920e37caa6a65c204cc478d8b80679c92581c09f1602647a not found: ID does not exist" containerID="534690997e7276a0920e37caa6a65c204cc478d8b80679c92581c09f1602647a"
Oct 30 01:22:03.742206 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:03.742117    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"534690997e7276a0920e37caa6a65c204cc478d8b80679c92581c09f1602647a"} err="failed to get container status \"534690997e7276a0920e37caa6a65c204cc478d8b80679c92581c09f1602647a\": rpc error: code = NotFound desc = could not find container \"534690997e7276a0920e37caa6a65c204cc478d8b80679c92581c09f1602647a\": container with ID starting with 534690997e7276a0920e37caa6a65c204cc478d8b80679c92581c09f1602647a not found: ID does not exist"
Oct 30 01:22:03.767227 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:03.767190    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-6f4fba-5c479c7dd7-9sxbh"]
Oct 30 01:22:03.770991 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:03.770962    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-6f4fba-5c479c7dd7-9sxbh"]
Oct 30 01:22:03.817982 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:03.817947    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-6f4fba-86877b49f8-n59lf"]
Oct 30 01:22:03.821568 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:03.821544    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-6f4fba-86877b49f8-n59lf"]
Oct 30 01:22:05.775770 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:05.774593    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd" path="/var/lib/kubelet/pods/2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd/volumes"
Oct 30 01:22:05.775770 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:05.775255    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44688868-ab51-4384-9887-b1d61810eca3" path="/var/lib/kubelet/pods/44688868-ab51-4384-9887-b1d61810eca3/volumes"
Oct 30 01:22:10.714137 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:10.714097    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8e0f66-8646cb7ccd-fw5mb"]
Oct 30 01:22:10.745864 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:10.745832    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e0f66-fc8567cdc-knftm"]
Oct 30 01:22:11.585295 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:11.585255    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv"]
Oct 30 01:22:11.585541 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:11.585515    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv" podUID="08c5615b-d854-4dd7-843e-9764dab5a127" containerName="maas-api" containerID="cri-o://c5b5b1cd585117146b93df782c21c9d0afd11d1bf3a84558455c9f2418ddf72d" gracePeriod=30
Oct 30 01:22:11.890136 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:11.890071    4393 generic.go:356] "Generic (PLEG): container finished" podID="08c5615b-d854-4dd7-843e-9764dab5a127" containerID="c5b5b1cd585117146b93df782c21c9d0afd11d1bf3a84558455c9f2418ddf72d" exitCode=0
Oct 30 01:22:11.890641 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:11.890467    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv" event={"ID":"08c5615b-d854-4dd7-843e-9764dab5a127","Type":"ContainerDied","Data":"c5b5b1cd585117146b93df782c21c9d0afd11d1bf3a84558455c9f2418ddf72d"}
Oct 30 01:22:11.908765 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:11.908730    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv"
Oct 30 01:22:11.911978 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:11.911944    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08c5615b-d854-4dd7-843e-9764dab5a127-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08c5615b-d854-4dd7-843e-9764dab5a127-openshift-service-ca\") pod \"08c5615b-d854-4dd7-843e-9764dab5a127\" (UID: \"08c5615b-d854-4dd7-843e-9764dab5a127\") "
Oct 30 01:22:11.912176 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:11.912064    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08c5615b-d854-4dd7-843e-9764dab5a127-kube-api-access-jzwr8\" (UniqueName: \"kubernetes.io/projected/08c5615b-d854-4dd7-843e-9764dab5a127-kube-api-access-jzwr8\") pod \"08c5615b-d854-4dd7-843e-9764dab5a127\" (UID: \"08c5615b-d854-4dd7-843e-9764dab5a127\") "
Oct 30 01:22:11.912176 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:11.912143    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08c5615b-d854-4dd7-843e-9764dab5a127-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08c5615b-d854-4dd7-843e-9764dab5a127-maas-api-tls\") pod \"08c5615b-d854-4dd7-843e-9764dab5a127\" (UID: \"08c5615b-d854-4dd7-843e-9764dab5a127\") "
Oct 30 01:22:11.912542 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:11.912349    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08c5615b-d854-4dd7-843e-9764dab5a127-openshift-service-ca" pod "08c5615b-d854-4dd7-843e-9764dab5a127" (UID: "08c5615b-d854-4dd7-843e-9764dab5a127"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:11.912943 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:11.912920    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08c5615b-d854-4dd7-843e-9764dab5a127-metrics-certs\" (UniqueName: \"kubernetes.io/secret/08c5615b-d854-4dd7-843e-9764dab5a127-metrics-certs\") pod \"08c5615b-d854-4dd7-843e-9764dab5a127\" (UID: \"08c5615b-d854-4dd7-843e-9764dab5a127\") "
Oct 30 01:22:11.913417 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:11.913398    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08c5615b-d854-4dd7-843e-9764dab5a127-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:11.914826 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:11.914797    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08c5615b-d854-4dd7-843e-9764dab5a127-maas-api-tls" pod "08c5615b-d854-4dd7-843e-9764dab5a127" (UID: "08c5615b-d854-4dd7-843e-9764dab5a127"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:11.915212 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:11.915185    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08c5615b-d854-4dd7-843e-9764dab5a127-metrics-certs" pod "08c5615b-d854-4dd7-843e-9764dab5a127" (UID: "08c5615b-d854-4dd7-843e-9764dab5a127"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:11.915534 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:11.915517    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c5615b-d854-4dd7-843e-9764dab5a127-kube-api-access-jzwr8" pod "08c5615b-d854-4dd7-843e-9764dab5a127" (UID: "08c5615b-d854-4dd7-843e-9764dab5a127"). InnerVolumeSpecName "kube-api-access-jzwr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:12.014988 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:12.014934    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzwr8\" (UniqueName: \"kubernetes.io/projected/08c5615b-d854-4dd7-843e-9764dab5a127-kube-api-access-jzwr8\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:12.014988 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:12.014976    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08c5615b-d854-4dd7-843e-9764dab5a127-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:12.014988 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:12.014993    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08c5615b-d854-4dd7-843e-9764dab5a127-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:12.899241 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:12.899205    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv"
Oct 30 01:22:12.899241 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:12.899233    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv" event={"ID":"08c5615b-d854-4dd7-843e-9764dab5a127","Type":"ContainerDied","Data":"1c94c833804b5899797c127a8610e83d004afa448970eb3ef954e68d583bb48b"}
Oct 30 01:22:12.899814 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:12.899299    4393 scope.go:122] "RemoveContainer" containerID="c5b5b1cd585117146b93df782c21c9d0afd11d1bf3a84558455c9f2418ddf72d"
Oct 30 01:22:12.950037 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:12.949998    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv"]
Oct 30 01:22:12.961422 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:12.961390    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-beeba4-5f7966cdbb-dv6xv"]
Oct 30 01:22:13.729832 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:13.729773    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08c5615b-d854-4dd7-843e-9764dab5a127" path="/var/lib/kubelet/pods/08c5615b-d854-4dd7-843e-9764dab5a127/volumes"
Oct 30 01:22:15.715205 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:15.715149    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e0f66-8646cb7ccd-fw5mb" podUID="3847bedb-934b-4dac-a5c1-7e5d593ba2ec" containerName="payload-processing" containerID="cri-o://1b3c77ad94471d5ade787d6731ff467e28ad8977ba5d80e6b7d6a8f7aa352d5b" gracePeriod=25
Oct 30 01:22:15.746905 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:15.746854    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e0f66-fc8567cdc-knftm" podUID="6bfbd1a2-2a38-43ed-abe0-c1a719454e10" containerName="payload-pre-processing" containerID="cri-o://a83eaebac0283dda84c234a6686f1efd9554b97a22c2b769d1b9e3136f65766a" gracePeriod=25
Oct 30 01:22:15.837124 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:15.837082    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf"]
Oct 30 01:22:15.837576 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:15.837521    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf" podUID="85f8ce06-4e78-43be-9c07-05c201eb70e8" containerName="maas-api" containerID="cri-o://1bb1a79e7160656e9c89168c4b69383ec83b6f7ab9e838b2705e67e911a90c60" gracePeriod=30
Oct 30 01:22:16.036852 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.036507    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-5f95998c75-v5lq9"]
Oct 30 01:22:16.037893 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.037845    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-5f95998c75-v5lq9" podUID="be58de6c-a4f3-40e5-9270-689a5eee91f1" containerName="payload-pre-processing" containerID="cri-o://1a58dfce66fd7c417df543896afe0ceb6bc1334ded8e5810fa058372f40d35cd" gracePeriod=30
Oct 30 01:22:16.177854 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.177820    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf"
Oct 30 01:22:16.197321 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.197288    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85f8ce06-4e78-43be-9c07-05c201eb70e8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f8ce06-4e78-43be-9c07-05c201eb70e8-openshift-service-ca\") pod \"85f8ce06-4e78-43be-9c07-05c201eb70e8\" (UID: \"85f8ce06-4e78-43be-9c07-05c201eb70e8\") "
Oct 30 01:22:16.197529 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.197387    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85f8ce06-4e78-43be-9c07-05c201eb70e8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f8ce06-4e78-43be-9c07-05c201eb70e8-maas-api-tls\") pod \"85f8ce06-4e78-43be-9c07-05c201eb70e8\" (UID: \"85f8ce06-4e78-43be-9c07-05c201eb70e8\") "
Oct 30 01:22:16.197529 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.197427    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85f8ce06-4e78-43be-9c07-05c201eb70e8-kube-api-access-59jjc\" (UniqueName: \"kubernetes.io/projected/85f8ce06-4e78-43be-9c07-05c201eb70e8-kube-api-access-59jjc\") pod \"85f8ce06-4e78-43be-9c07-05c201eb70e8\" (UID: \"85f8ce06-4e78-43be-9c07-05c201eb70e8\") "
Oct 30 01:22:16.197529 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.197462    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85f8ce06-4e78-43be-9c07-05c201eb70e8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f8ce06-4e78-43be-9c07-05c201eb70e8-metrics-certs\") pod \"85f8ce06-4e78-43be-9c07-05c201eb70e8\" (UID: \"85f8ce06-4e78-43be-9c07-05c201eb70e8\") "
Oct 30 01:22:16.198637 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.198603    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85f8ce06-4e78-43be-9c07-05c201eb70e8-openshift-service-ca" pod "85f8ce06-4e78-43be-9c07-05c201eb70e8" (UID: "85f8ce06-4e78-43be-9c07-05c201eb70e8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:16.200818 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.200733    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f8ce06-4e78-43be-9c07-05c201eb70e8-metrics-certs" pod "85f8ce06-4e78-43be-9c07-05c201eb70e8" (UID: "85f8ce06-4e78-43be-9c07-05c201eb70e8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:16.200818 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.200752    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f8ce06-4e78-43be-9c07-05c201eb70e8-maas-api-tls" pod "85f8ce06-4e78-43be-9c07-05c201eb70e8" (UID: "85f8ce06-4e78-43be-9c07-05c201eb70e8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:16.202530 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.202506    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f8ce06-4e78-43be-9c07-05c201eb70e8-kube-api-access-59jjc" pod "85f8ce06-4e78-43be-9c07-05c201eb70e8" (UID: "85f8ce06-4e78-43be-9c07-05c201eb70e8"). InnerVolumeSpecName "kube-api-access-59jjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:16.299607 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.299513    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f8ce06-4e78-43be-9c07-05c201eb70e8-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:16.299607 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.299546    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f8ce06-4e78-43be-9c07-05c201eb70e8-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:16.299607 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.299557    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59jjc\" (UniqueName: \"kubernetes.io/projected/85f8ce06-4e78-43be-9c07-05c201eb70e8-kube-api-access-59jjc\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:16.299607 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.299565    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f8ce06-4e78-43be-9c07-05c201eb70e8-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:16.313441 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.313412    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-5f95998c75-v5lq9"
Oct 30 01:22:16.401001 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.400619    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be58de6c-a4f3-40e5-9270-689a5eee91f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be58de6c-a4f3-40e5-9270-689a5eee91f1-plugins-config-volume\") pod \"be58de6c-a4f3-40e5-9270-689a5eee91f1\" (UID: \"be58de6c-a4f3-40e5-9270-689a5eee91f1\") "
Oct 30 01:22:16.401236 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.401005    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be58de6c-a4f3-40e5-9270-689a5eee91f1-kube-api-access-s5ckr\" (UniqueName: \"kubernetes.io/projected/be58de6c-a4f3-40e5-9270-689a5eee91f1-kube-api-access-s5ckr\") pod \"be58de6c-a4f3-40e5-9270-689a5eee91f1\" (UID: \"be58de6c-a4f3-40e5-9270-689a5eee91f1\") "
Oct 30 01:22:16.401904 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.401876    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be58de6c-a4f3-40e5-9270-689a5eee91f1-plugins-config-volume" pod "be58de6c-a4f3-40e5-9270-689a5eee91f1" (UID: "be58de6c-a4f3-40e5-9270-689a5eee91f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:16.410303 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.407629    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be58de6c-a4f3-40e5-9270-689a5eee91f1-kube-api-access-s5ckr" pod "be58de6c-a4f3-40e5-9270-689a5eee91f1" (UID: "be58de6c-a4f3-40e5-9270-689a5eee91f1"). InnerVolumeSpecName "kube-api-access-s5ckr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:16.503118 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.503075    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5ckr\" (UniqueName: \"kubernetes.io/projected/be58de6c-a4f3-40e5-9270-689a5eee91f1-kube-api-access-s5ckr\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:16.503118 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.503109    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be58de6c-a4f3-40e5-9270-689a5eee91f1-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:16.930632 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.930591    4393 generic.go:356] "Generic (PLEG): container finished" podID="be58de6c-a4f3-40e5-9270-689a5eee91f1" containerID="1a58dfce66fd7c417df543896afe0ceb6bc1334ded8e5810fa058372f40d35cd" exitCode=0
Oct 30 01:22:16.931120 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.930647    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-5f95998c75-v5lq9"
Oct 30 01:22:16.931120 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.930676    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-5f95998c75-v5lq9" event={"ID":"be58de6c-a4f3-40e5-9270-689a5eee91f1","Type":"ContainerDied","Data":"1a58dfce66fd7c417df543896afe0ceb6bc1334ded8e5810fa058372f40d35cd"}
Oct 30 01:22:16.931120 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.930739    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-5f95998c75-v5lq9" event={"ID":"be58de6c-a4f3-40e5-9270-689a5eee91f1","Type":"ContainerDied","Data":"dcdc97a089365076aaf3051b4fe62158290752af9703fb786f31a9f410fcb62c"}
Oct 30 01:22:16.931120 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.930764    4393 scope.go:122] "RemoveContainer" containerID="1a58dfce66fd7c417df543896afe0ceb6bc1334ded8e5810fa058372f40d35cd"
Oct 30 01:22:16.932694 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.932652    4393 generic.go:356] "Generic (PLEG): container finished" podID="85f8ce06-4e78-43be-9c07-05c201eb70e8" containerID="1bb1a79e7160656e9c89168c4b69383ec83b6f7ab9e838b2705e67e911a90c60" exitCode=0
Oct 30 01:22:16.932694 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.932694    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf"
Oct 30 01:22:16.932920 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.932726    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf" event={"ID":"85f8ce06-4e78-43be-9c07-05c201eb70e8","Type":"ContainerDied","Data":"1bb1a79e7160656e9c89168c4b69383ec83b6f7ab9e838b2705e67e911a90c60"}
Oct 30 01:22:16.932920 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.932767    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf" event={"ID":"85f8ce06-4e78-43be-9c07-05c201eb70e8","Type":"ContainerDied","Data":"f71f51023409e8d9ffc53a040bdff3485b4a72d42dc689cdd63b7a3d84a0346c"}
Oct 30 01:22:16.944417 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.944398    4393 scope.go:122] "RemoveContainer" containerID="1a58dfce66fd7c417df543896afe0ceb6bc1334ded8e5810fa058372f40d35cd"
Oct 30 01:22:16.944809 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:22:16.944756    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a58dfce66fd7c417df543896afe0ceb6bc1334ded8e5810fa058372f40d35cd\": container with ID starting with 1a58dfce66fd7c417df543896afe0ceb6bc1334ded8e5810fa058372f40d35cd not found: ID does not exist" containerID="1a58dfce66fd7c417df543896afe0ceb6bc1334ded8e5810fa058372f40d35cd"
Oct 30 01:22:16.944976 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.944820    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a58dfce66fd7c417df543896afe0ceb6bc1334ded8e5810fa058372f40d35cd"} err="failed to get container status \"1a58dfce66fd7c417df543896afe0ceb6bc1334ded8e5810fa058372f40d35cd\": rpc error: code = NotFound desc = could not find container \"1a58dfce66fd7c417df543896afe0ceb6bc1334ded8e5810fa058372f40d35cd\": container with ID starting with 1a58dfce66fd7c417df543896afe0ceb6bc1334ded8e5810fa058372f40d35cd not found: ID does not exist"
Oct 30 01:22:16.944976 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.944839    4393 scope.go:122] "RemoveContainer" containerID="1bb1a79e7160656e9c89168c4b69383ec83b6f7ab9e838b2705e67e911a90c60"
Oct 30 01:22:16.955356 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.955330    4393 scope.go:122] "RemoveContainer" containerID="1bb1a79e7160656e9c89168c4b69383ec83b6f7ab9e838b2705e67e911a90c60"
Oct 30 01:22:16.955679 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:22:16.955657    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bb1a79e7160656e9c89168c4b69383ec83b6f7ab9e838b2705e67e911a90c60\": container with ID starting with 1bb1a79e7160656e9c89168c4b69383ec83b6f7ab9e838b2705e67e911a90c60 not found: ID does not exist" containerID="1bb1a79e7160656e9c89168c4b69383ec83b6f7ab9e838b2705e67e911a90c60"
Oct 30 01:22:16.955764 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.955692    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bb1a79e7160656e9c89168c4b69383ec83b6f7ab9e838b2705e67e911a90c60"} err="failed to get container status \"1bb1a79e7160656e9c89168c4b69383ec83b6f7ab9e838b2705e67e911a90c60\": rpc error: code = NotFound desc = could not find container \"1bb1a79e7160656e9c89168c4b69383ec83b6f7ab9e838b2705e67e911a90c60\": container with ID starting with 1bb1a79e7160656e9c89168c4b69383ec83b6f7ab9e838b2705e67e911a90c60 not found: ID does not exist"
Oct 30 01:22:16.981650 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.981615    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf"]
Oct 30 01:22:16.987039 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:16.987006    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-812ba2-b4f94bff7-pc4cf"]
Oct 30 01:22:17.020088 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:17.020045    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-5f95998c75-v5lq9"]
Oct 30 01:22:17.024442 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:17.024397    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-812ba2-5f95998c75-v5lq9"]
Oct 30 01:22:17.724962 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:17.724927    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85f8ce06-4e78-43be-9c07-05c201eb70e8" path="/var/lib/kubelet/pods/85f8ce06-4e78-43be-9c07-05c201eb70e8/volumes"
Oct 30 01:22:17.725383 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:17.725360    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be58de6c-a4f3-40e5-9270-689a5eee91f1" path="/var/lib/kubelet/pods/be58de6c-a4f3-40e5-9270-689a5eee91f1/volumes"
Oct 30 01:22:19.311139 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:19.311100    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"]
Oct 30 01:22:19.311662 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:19.311394    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8" podUID="40a26137-dc63-40e1-89cc-2a0c70d6f52e" containerName="istio-proxy" containerID="cri-o://ef2bacbded603aa589533f089fe22a99843fd42bda70befcccd7934d258d89cd" gracePeriod=30
Oct 30 01:22:23.606594 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:23.606539    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e0f66-8646cb7ccd-fw5mb" podUID="3847bedb-934b-4dac-a5c1-7e5d593ba2ec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:22:23.607173 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:23.606539    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e0f66-fc8567cdc-knftm" podUID="6bfbd1a2-2a38-43ed-abe0-c1a719454e10" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:22:24.601232 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:24.601155    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"
Oct 30 01:22:24.723086 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:24.723041    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40a26137-dc63-40e1-89cc-2a0c70d6f52e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a26137-dc63-40e1-89cc-2a0c70d6f52e-istio-podinfo\") pod \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\" (UID: \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\") "
Oct 30 01:22:24.723681 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:24.723223    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a26137-dc63-40e1-89cc-2a0c70d6f52e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a26137-dc63-40e1-89cc-2a0c70d6f52e-istio-envoy\") pod \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\" (UID: \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\") "
Oct 30 01:22:24.723681 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:24.723282    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40a26137-dc63-40e1-89cc-2a0c70d6f52e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a26137-dc63-40e1-89cc-2a0c70d6f52e-istiod-ca-cert\") pod \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\" (UID: \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\") "
Oct 30 01:22:24.723681 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:24.723363    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a26137-dc63-40e1-89cc-2a0c70d6f52e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a26137-dc63-40e1-89cc-2a0c70d6f52e-credential-socket\") pod \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\" (UID: \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\") "
Oct 30 01:22:24.723681 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:24.723405    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40a26137-dc63-40e1-89cc-2a0c70d6f52e-istio-token\" (UniqueName: \"kubernetes.io/projected/40a26137-dc63-40e1-89cc-2a0c70d6f52e-istio-token\") pod \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\" (UID: \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\") "
Oct 30 01:22:24.723681 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:24.723439    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a26137-dc63-40e1-89cc-2a0c70d6f52e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a26137-dc63-40e1-89cc-2a0c70d6f52e-workload-certs\") pod \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\" (UID: \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\") "
Oct 30 01:22:24.723681 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:24.723514    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a26137-dc63-40e1-89cc-2a0c70d6f52e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a26137-dc63-40e1-89cc-2a0c70d6f52e-istio-data\") pod \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\" (UID: \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\") "
Oct 30 01:22:24.723681 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:24.723588    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40a26137-dc63-40e1-89cc-2a0c70d6f52e-kube-api-access-29cm6\" (UniqueName: \"kubernetes.io/projected/40a26137-dc63-40e1-89cc-2a0c70d6f52e-kube-api-access-29cm6\") pod \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\" (UID: \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\") "
Oct 30 01:22:24.723681 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:24.723617    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a26137-dc63-40e1-89cc-2a0c70d6f52e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a26137-dc63-40e1-89cc-2a0c70d6f52e-workload-socket\") pod \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\" (UID: \"40a26137-dc63-40e1-89cc-2a0c70d6f52e\") "
Oct 30 01:22:24.724237 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:24.723676    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a26137-dc63-40e1-89cc-2a0c70d6f52e-credential-socket" pod "40a26137-dc63-40e1-89cc-2a0c70d6f52e" (UID: "40a26137-dc63-40e1-89cc-2a0c70d6f52e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:24.724237 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:24.723837    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40a26137-dc63-40e1-89cc-2a0c70d6f52e-istiod-ca-cert" pod "40a26137-dc63-40e1-89cc-2a0c70d6f52e" (UID: "40a26137-dc63-40e1-89cc-2a0c70d6f52e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:24.724237 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:24.723920    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a26137-dc63-40e1-89cc-2a0c70d6f52e-istio-data" pod "40a26137-dc63-40e1-89cc-2a0c70d6f52e" (UID: "40a26137-dc63-40e1-89cc-2a0c70d6f52e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:24.724237 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:24.724114    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a26137-dc63-40e1-89cc-2a0c70d6f52e-workload-socket" pod "40a26137-dc63-40e1-89cc-2a0c70d6f52e" (UID: "40a26137-dc63-40e1-89cc-2a0c70d6f52e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:24.724379 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:24.724261    4393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a26137-dc63-40e1-89cc-2a0c70d6f52e-workload-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:24.724379 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:24.724274    4393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a26137-dc63-40e1-89cc-2a0c70d6f52e-istiod-ca-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:24.724379 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:24.724283    4393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a26137-dc63-40e1-89cc-2a0c70d6f52e-credential-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:24.724379 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:24.724292    4393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a26137-dc63-40e1-89cc-2a0c70d6f52e-istio-data\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:24.724379 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:24.724127    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a26137-dc63-40e1-89cc-2a0c70d6f52e-workload-certs" pod "40a26137-dc63-40e1-89cc-2a0c70d6f52e" (UID: "40a26137-dc63-40e1-89cc-2a0c70d6f52e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:24.725890 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:24.725865    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40a26137-dc63-40e1-89cc-2a0c70d6f52e-istio-token" pod "40a26137-dc63-40e1-89cc-2a0c70d6f52e" (UID: "40a26137-dc63-40e1-89cc-2a0c70d6f52e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:24.726071 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:24.726054    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40a26137-dc63-40e1-89cc-2a0c70d6f52e-istio-podinfo" pod "40a26137-dc63-40e1-89cc-2a0c70d6f52e" (UID: "40a26137-dc63-40e1-89cc-2a0c70d6f52e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:22:24.726184 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:24.726165    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a26137-dc63-40e1-89cc-2a0c70d6f52e-istio-envoy" pod "40a26137-dc63-40e1-89cc-2a0c70d6f52e" (UID: "40a26137-dc63-40e1-89cc-2a0c70d6f52e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:22:24.726568 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:24.726544    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40a26137-dc63-40e1-89cc-2a0c70d6f52e-kube-api-access-29cm6" pod "40a26137-dc63-40e1-89cc-2a0c70d6f52e" (UID: "40a26137-dc63-40e1-89cc-2a0c70d6f52e"). InnerVolumeSpecName "kube-api-access-29cm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:24.825536 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:24.825498    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29cm6\" (UniqueName: \"kubernetes.io/projected/40a26137-dc63-40e1-89cc-2a0c70d6f52e-kube-api-access-29cm6\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:24.825536 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:24.825531    4393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a26137-dc63-40e1-89cc-2a0c70d6f52e-istio-podinfo\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:24.825536 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:24.825541    4393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a26137-dc63-40e1-89cc-2a0c70d6f52e-istio-envoy\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:24.825925 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:24.825550    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40a26137-dc63-40e1-89cc-2a0c70d6f52e-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:24.825925 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:24.825560    4393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a26137-dc63-40e1-89cc-2a0c70d6f52e-workload-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:24.993838 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:24.993799    4393 generic.go:356] "Generic (PLEG): container finished" podID="40a26137-dc63-40e1-89cc-2a0c70d6f52e" containerID="ef2bacbded603aa589533f089fe22a99843fd42bda70befcccd7934d258d89cd" exitCode=0
Oct 30 01:22:24.994045 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:24.993850    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"
Oct 30 01:22:24.994045 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:24.993867    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8" event={"ID":"40a26137-dc63-40e1-89cc-2a0c70d6f52e","Type":"ContainerDied","Data":"ef2bacbded603aa589533f089fe22a99843fd42bda70befcccd7934d258d89cd"}
Oct 30 01:22:24.994045 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:24.993924    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8" event={"ID":"40a26137-dc63-40e1-89cc-2a0c70d6f52e","Type":"ContainerDied","Data":"016f617a4502c4dfb77692a0d4ffd5192a23a498c25491be1be6a178387e2f81"}
Oct 30 01:22:24.994045 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:24.993950    4393 scope.go:122] "RemoveContainer" containerID="ef2bacbded603aa589533f089fe22a99843fd42bda70befcccd7934d258d89cd"
Oct 30 01:22:25.005962 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:25.005943    4393 scope.go:122] "RemoveContainer" containerID="ef2bacbded603aa589533f089fe22a99843fd42bda70befcccd7934d258d89cd"
Oct 30 01:22:25.006258 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:22:25.006237    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef2bacbded603aa589533f089fe22a99843fd42bda70befcccd7934d258d89cd\": container with ID starting with ef2bacbded603aa589533f089fe22a99843fd42bda70befcccd7934d258d89cd not found: ID does not exist" containerID="ef2bacbded603aa589533f089fe22a99843fd42bda70befcccd7934d258d89cd"
Oct 30 01:22:25.006303 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:25.006270    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef2bacbded603aa589533f089fe22a99843fd42bda70befcccd7934d258d89cd"} err="failed to get container status \"ef2bacbded603aa589533f089fe22a99843fd42bda70befcccd7934d258d89cd\": rpc error: code = NotFound desc = could not find container \"ef2bacbded603aa589533f089fe22a99843fd42bda70befcccd7934d258d89cd\": container with ID starting with ef2bacbded603aa589533f089fe22a99843fd42bda70befcccd7934d258d89cd not found: ID does not exist"
Oct 30 01:22:25.045128 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:25.045094    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"]
Oct 30 01:22:25.056154 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:25.056122    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-beeba4-openshift-default-6ff99b6957-xkhd8"]
Oct 30 01:22:25.726870 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:25.726831    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40a26137-dc63-40e1-89cc-2a0c70d6f52e" path="/var/lib/kubelet/pods/40a26137-dc63-40e1-89cc-2a0c70d6f52e/volumes"
Oct 30 01:22:26.513455 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:22:26.513414    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:22:33.606996 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:33.606955    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e0f66-8646cb7ccd-fw5mb" podUID="3847bedb-934b-4dac-a5c1-7e5d593ba2ec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:22:33.606996 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:33.606979    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e0f66-fc8567cdc-knftm" podUID="6bfbd1a2-2a38-43ed-abe0-c1a719454e10" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:22:36.029607 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:36.029574    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e0f66-8646cb7ccd-fw5mb"
Oct 30 01:22:36.078583 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:36.078539    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e0f66-fc8567cdc-knftm"
Oct 30 01:22:36.079219 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:36.079194    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3847bedb-934b-4dac-a5c1-7e5d593ba2ec-kube-api-access-67jrz\" (UniqueName: \"kubernetes.io/projected/3847bedb-934b-4dac-a5c1-7e5d593ba2ec-kube-api-access-67jrz\") pod \"3847bedb-934b-4dac-a5c1-7e5d593ba2ec\" (UID: \"3847bedb-934b-4dac-a5c1-7e5d593ba2ec\") "
Oct 30 01:22:36.079468 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:36.079359    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3847bedb-934b-4dac-a5c1-7e5d593ba2ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3847bedb-934b-4dac-a5c1-7e5d593ba2ec-plugins-config-volume\") pod \"3847bedb-934b-4dac-a5c1-7e5d593ba2ec\" (UID: \"3847bedb-934b-4dac-a5c1-7e5d593ba2ec\") "
Oct 30 01:22:36.079747 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:36.079720    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3847bedb-934b-4dac-a5c1-7e5d593ba2ec-plugins-config-volume" pod "3847bedb-934b-4dac-a5c1-7e5d593ba2ec" (UID: "3847bedb-934b-4dac-a5c1-7e5d593ba2ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:36.080621 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:36.080598    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3847bedb-934b-4dac-a5c1-7e5d593ba2ec-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:36.082178 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:36.082155    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3847bedb-934b-4dac-a5c1-7e5d593ba2ec-kube-api-access-67jrz" pod "3847bedb-934b-4dac-a5c1-7e5d593ba2ec" (UID: "3847bedb-934b-4dac-a5c1-7e5d593ba2ec"). InnerVolumeSpecName "kube-api-access-67jrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:36.082276 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:36.082248    4393 generic.go:356] "Generic (PLEG): container finished" podID="3847bedb-934b-4dac-a5c1-7e5d593ba2ec" containerID="1b3c77ad94471d5ade787d6731ff467e28ad8977ba5d80e6b7d6a8f7aa352d5b" exitCode=0
Oct 30 01:22:36.082322 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:36.082296    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e0f66-8646cb7ccd-fw5mb"
Oct 30 01:22:36.082355 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:36.082327    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e0f66-8646cb7ccd-fw5mb" event={"ID":"3847bedb-934b-4dac-a5c1-7e5d593ba2ec","Type":"ContainerDied","Data":"1b3c77ad94471d5ade787d6731ff467e28ad8977ba5d80e6b7d6a8f7aa352d5b"}
Oct 30 01:22:36.082393 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:36.082380    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8e0f66-8646cb7ccd-fw5mb" event={"ID":"3847bedb-934b-4dac-a5c1-7e5d593ba2ec","Type":"ContainerDied","Data":"e87b18ce6c1a24ba6268cafb05ca601ee9723763381d56e3ebad91b8b00460ae"}
Oct 30 01:22:36.082447 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:36.082410    4393 scope.go:122] "RemoveContainer" containerID="1b3c77ad94471d5ade787d6731ff467e28ad8977ba5d80e6b7d6a8f7aa352d5b"
Oct 30 01:22:36.091947 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:36.091908    4393 generic.go:356] "Generic (PLEG): container finished" podID="6bfbd1a2-2a38-43ed-abe0-c1a719454e10" containerID="a83eaebac0283dda84c234a6686f1efd9554b97a22c2b769d1b9e3136f65766a" exitCode=0
Oct 30 01:22:36.092089 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:36.092028    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e0f66-fc8567cdc-knftm" event={"ID":"6bfbd1a2-2a38-43ed-abe0-c1a719454e10","Type":"ContainerDied","Data":"a83eaebac0283dda84c234a6686f1efd9554b97a22c2b769d1b9e3136f65766a"}
Oct 30 01:22:36.092137 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:36.092120    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e0f66-fc8567cdc-knftm" event={"ID":"6bfbd1a2-2a38-43ed-abe0-c1a719454e10","Type":"ContainerDied","Data":"769df99ac61d10fb86be021f80b171d3a7fb1e5aea5ef93ac046a20c38c80acb"}
Oct 30 01:22:36.092227 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:36.092211    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e0f66-fc8567cdc-knftm"
Oct 30 01:22:36.102505 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:36.102480    4393 scope.go:122] "RemoveContainer" containerID="1b3c77ad94471d5ade787d6731ff467e28ad8977ba5d80e6b7d6a8f7aa352d5b"
Oct 30 01:22:36.102897 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:22:36.102871    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b3c77ad94471d5ade787d6731ff467e28ad8977ba5d80e6b7d6a8f7aa352d5b\": container with ID starting with 1b3c77ad94471d5ade787d6731ff467e28ad8977ba5d80e6b7d6a8f7aa352d5b not found: ID does not exist" containerID="1b3c77ad94471d5ade787d6731ff467e28ad8977ba5d80e6b7d6a8f7aa352d5b"
Oct 30 01:22:36.103008 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:36.102905    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b3c77ad94471d5ade787d6731ff467e28ad8977ba5d80e6b7d6a8f7aa352d5b"} err="failed to get container status \"1b3c77ad94471d5ade787d6731ff467e28ad8977ba5d80e6b7d6a8f7aa352d5b\": rpc error: code = NotFound desc = could not find container \"1b3c77ad94471d5ade787d6731ff467e28ad8977ba5d80e6b7d6a8f7aa352d5b\": container with ID starting with 1b3c77ad94471d5ade787d6731ff467e28ad8977ba5d80e6b7d6a8f7aa352d5b not found: ID does not exist"
Oct 30 01:22:36.103008 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:36.102929    4393 scope.go:122] "RemoveContainer" containerID="a83eaebac0283dda84c234a6686f1efd9554b97a22c2b769d1b9e3136f65766a"
Oct 30 01:22:36.113880 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:36.113859    4393 scope.go:122] "RemoveContainer" containerID="a83eaebac0283dda84c234a6686f1efd9554b97a22c2b769d1b9e3136f65766a"
Oct 30 01:22:36.114233 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:22:36.114210    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a83eaebac0283dda84c234a6686f1efd9554b97a22c2b769d1b9e3136f65766a\": container with ID starting with a83eaebac0283dda84c234a6686f1efd9554b97a22c2b769d1b9e3136f65766a not found: ID does not exist" containerID="a83eaebac0283dda84c234a6686f1efd9554b97a22c2b769d1b9e3136f65766a"
Oct 30 01:22:36.114308 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:36.114244    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a83eaebac0283dda84c234a6686f1efd9554b97a22c2b769d1b9e3136f65766a"} err="failed to get container status \"a83eaebac0283dda84c234a6686f1efd9554b97a22c2b769d1b9e3136f65766a\": rpc error: code = NotFound desc = could not find container \"a83eaebac0283dda84c234a6686f1efd9554b97a22c2b769d1b9e3136f65766a\": container with ID starting with a83eaebac0283dda84c234a6686f1efd9554b97a22c2b769d1b9e3136f65766a not found: ID does not exist"
Oct 30 01:22:36.155231 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:36.155195    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8e0f66-8646cb7ccd-fw5mb"]
Oct 30 01:22:36.158889 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:36.158861    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8e0f66-8646cb7ccd-fw5mb"]
Oct 30 01:22:36.181432 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:36.181400    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bfbd1a2-2a38-43ed-abe0-c1a719454e10-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bfbd1a2-2a38-43ed-abe0-c1a719454e10-plugins-config-volume\") pod \"6bfbd1a2-2a38-43ed-abe0-c1a719454e10\" (UID: \"6bfbd1a2-2a38-43ed-abe0-c1a719454e10\") "
Oct 30 01:22:36.181615 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:36.181510    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bfbd1a2-2a38-43ed-abe0-c1a719454e10-kube-api-access-4cxc5\" (UniqueName: \"kubernetes.io/projected/6bfbd1a2-2a38-43ed-abe0-c1a719454e10-kube-api-access-4cxc5\") pod \"6bfbd1a2-2a38-43ed-abe0-c1a719454e10\" (UID: \"6bfbd1a2-2a38-43ed-abe0-c1a719454e10\") "
Oct 30 01:22:36.181866 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:36.181842    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bfbd1a2-2a38-43ed-abe0-c1a719454e10-plugins-config-volume" pod "6bfbd1a2-2a38-43ed-abe0-c1a719454e10" (UID: "6bfbd1a2-2a38-43ed-abe0-c1a719454e10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:36.182110 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:36.182094    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67jrz\" (UniqueName: \"kubernetes.io/projected/3847bedb-934b-4dac-a5c1-7e5d593ba2ec-kube-api-access-67jrz\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:36.182172 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:36.182113    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bfbd1a2-2a38-43ed-abe0-c1a719454e10-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:36.183941 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:36.183919    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bfbd1a2-2a38-43ed-abe0-c1a719454e10-kube-api-access-4cxc5" pod "6bfbd1a2-2a38-43ed-abe0-c1a719454e10" (UID: "6bfbd1a2-2a38-43ed-abe0-c1a719454e10"). InnerVolumeSpecName "kube-api-access-4cxc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:36.283231 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:36.283148    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cxc5\" (UniqueName: \"kubernetes.io/projected/6bfbd1a2-2a38-43ed-abe0-c1a719454e10-kube-api-access-4cxc5\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:36.437664 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:36.437628    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e0f66-fc8567cdc-knftm"]
Oct 30 01:22:36.442878 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:36.442848    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8e0f66-fc8567cdc-knftm"]
Oct 30 01:22:37.724976 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:37.724941    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3847bedb-934b-4dac-a5c1-7e5d593ba2ec" path="/var/lib/kubelet/pods/3847bedb-934b-4dac-a5c1-7e5d593ba2ec/volumes"
Oct 30 01:22:37.725398 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:37.725383    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bfbd1a2-2a38-43ed-abe0-c1a719454e10" path="/var/lib/kubelet/pods/6bfbd1a2-2a38-43ed-abe0-c1a719454e10/volumes"
Oct 30 01:22:38.780379 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:38.780348    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx"]
Oct 30 01:22:38.780912 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:38.780612    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx" podUID="7416ca5a-6ff8-4937-84cf-15a156c523eb" containerName="maas-api" containerID="cri-o://b92dee2a8d518ec864eaa875cf36a3210af817ca720fef3515a0745790c3e17a" gracePeriod=30
Oct 30 01:22:39.086952 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:39.086921    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx"
Oct 30 01:22:39.119590 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:39.119552    4393 generic.go:356] "Generic (PLEG): container finished" podID="7416ca5a-6ff8-4937-84cf-15a156c523eb" containerID="b92dee2a8d518ec864eaa875cf36a3210af817ca720fef3515a0745790c3e17a" exitCode=0
Oct 30 01:22:39.119753 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:39.119599    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx"
Oct 30 01:22:39.119753 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:39.119623    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx" event={"ID":"7416ca5a-6ff8-4937-84cf-15a156c523eb","Type":"ContainerDied","Data":"b92dee2a8d518ec864eaa875cf36a3210af817ca720fef3515a0745790c3e17a"}
Oct 30 01:22:39.119753 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:39.119678    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx" event={"ID":"7416ca5a-6ff8-4937-84cf-15a156c523eb","Type":"ContainerDied","Data":"3ae242af21d20f74af14a1d7d3bc3a9e62e73bd6143d348b0d6f09103ae26479"}
Oct 30 01:22:39.119753 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:39.119699    4393 scope.go:122] "RemoveContainer" containerID="b92dee2a8d518ec864eaa875cf36a3210af817ca720fef3515a0745790c3e17a"
Oct 30 01:22:39.130985 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:39.127768    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7416ca5a-6ff8-4937-84cf-15a156c523eb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7416ca5a-6ff8-4937-84cf-15a156c523eb-openshift-service-ca\") pod \"7416ca5a-6ff8-4937-84cf-15a156c523eb\" (UID: \"7416ca5a-6ff8-4937-84cf-15a156c523eb\") "
Oct 30 01:22:39.130985 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:39.127836    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7416ca5a-6ff8-4937-84cf-15a156c523eb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7416ca5a-6ff8-4937-84cf-15a156c523eb-maas-api-tls\") pod \"7416ca5a-6ff8-4937-84cf-15a156c523eb\" (UID: \"7416ca5a-6ff8-4937-84cf-15a156c523eb\") "
Oct 30 01:22:39.130985 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:39.127889    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7416ca5a-6ff8-4937-84cf-15a156c523eb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7416ca5a-6ff8-4937-84cf-15a156c523eb-metrics-certs\") pod \"7416ca5a-6ff8-4937-84cf-15a156c523eb\" (UID: \"7416ca5a-6ff8-4937-84cf-15a156c523eb\") "
Oct 30 01:22:39.130985 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:39.128133    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7416ca5a-6ff8-4937-84cf-15a156c523eb-kube-api-access-ktfrn\" (UniqueName: \"kubernetes.io/projected/7416ca5a-6ff8-4937-84cf-15a156c523eb-kube-api-access-ktfrn\") pod \"7416ca5a-6ff8-4937-84cf-15a156c523eb\" (UID: \"7416ca5a-6ff8-4937-84cf-15a156c523eb\") "
Oct 30 01:22:39.130985 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:39.129941    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7416ca5a-6ff8-4937-84cf-15a156c523eb-openshift-service-ca" pod "7416ca5a-6ff8-4937-84cf-15a156c523eb" (UID: "7416ca5a-6ff8-4937-84cf-15a156c523eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:39.131360 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:39.131299    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7416ca5a-6ff8-4937-84cf-15a156c523eb-kube-api-access-ktfrn" pod "7416ca5a-6ff8-4937-84cf-15a156c523eb" (UID: "7416ca5a-6ff8-4937-84cf-15a156c523eb"). InnerVolumeSpecName "kube-api-access-ktfrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:39.131926 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:39.131898    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7416ca5a-6ff8-4937-84cf-15a156c523eb-maas-api-tls" pod "7416ca5a-6ff8-4937-84cf-15a156c523eb" (UID: "7416ca5a-6ff8-4937-84cf-15a156c523eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:39.134964 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:39.133337    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7416ca5a-6ff8-4937-84cf-15a156c523eb-metrics-certs" pod "7416ca5a-6ff8-4937-84cf-15a156c523eb" (UID: "7416ca5a-6ff8-4937-84cf-15a156c523eb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:39.134964 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:39.133460    4393 scope.go:122] "RemoveContainer" containerID="b92dee2a8d518ec864eaa875cf36a3210af817ca720fef3515a0745790c3e17a"
Oct 30 01:22:39.134964 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:22:39.133841    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b92dee2a8d518ec864eaa875cf36a3210af817ca720fef3515a0745790c3e17a\": container with ID starting with b92dee2a8d518ec864eaa875cf36a3210af817ca720fef3515a0745790c3e17a not found: ID does not exist" containerID="b92dee2a8d518ec864eaa875cf36a3210af817ca720fef3515a0745790c3e17a"
Oct 30 01:22:39.134964 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:39.133877    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b92dee2a8d518ec864eaa875cf36a3210af817ca720fef3515a0745790c3e17a"} err="failed to get container status \"b92dee2a8d518ec864eaa875cf36a3210af817ca720fef3515a0745790c3e17a\": rpc error: code = NotFound desc = could not find container \"b92dee2a8d518ec864eaa875cf36a3210af817ca720fef3515a0745790c3e17a\": container with ID starting with b92dee2a8d518ec864eaa875cf36a3210af817ca720fef3515a0745790c3e17a not found: ID does not exist"
Oct 30 01:22:39.230063 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:39.230031    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktfrn\" (UniqueName: \"kubernetes.io/projected/7416ca5a-6ff8-4937-84cf-15a156c523eb-kube-api-access-ktfrn\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:39.230063 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:39.230060    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7416ca5a-6ff8-4937-84cf-15a156c523eb-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:39.230063 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:39.230072    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7416ca5a-6ff8-4937-84cf-15a156c523eb-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:39.230063 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:39.230081    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7416ca5a-6ff8-4937-84cf-15a156c523eb-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:39.467796 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:39.467756    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx"]
Oct 30 01:22:39.478627 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:39.478590    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8e0f66-77795dbd5d-8tftx"]
Oct 30 01:22:39.725182 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:39.725084    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7416ca5a-6ff8-4937-84cf-15a156c523eb" path="/var/lib/kubelet/pods/7416ca5a-6ff8-4937-84cf-15a156c523eb/volumes"
Oct 30 01:22:41.602137 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:41.602099    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm"]
Oct 30 01:22:41.602619 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:41.602341    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm" podUID="2526f9aa-9b27-4a09-b321-ec07b6b0131e" containerName="maas-api" containerID="cri-o://9363b2c1be68aada576269291ab458aad0361a17baeb10a5494bd9faf6179fd7" gracePeriod=30
Oct 30 01:22:41.916434 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:41.916408    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm"
Oct 30 01:22:41.967956 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:41.967918    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2526f9aa-9b27-4a09-b321-ec07b6b0131e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2526f9aa-9b27-4a09-b321-ec07b6b0131e-openshift-service-ca\") pod \"2526f9aa-9b27-4a09-b321-ec07b6b0131e\" (UID: \"2526f9aa-9b27-4a09-b321-ec07b6b0131e\") "
Oct 30 01:22:41.968422 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:41.968017    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2526f9aa-9b27-4a09-b321-ec07b6b0131e-kube-api-access-xl9kg\" (UniqueName: \"kubernetes.io/projected/2526f9aa-9b27-4a09-b321-ec07b6b0131e-kube-api-access-xl9kg\") pod \"2526f9aa-9b27-4a09-b321-ec07b6b0131e\" (UID: \"2526f9aa-9b27-4a09-b321-ec07b6b0131e\") "
Oct 30 01:22:41.968422 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:41.968080    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2526f9aa-9b27-4a09-b321-ec07b6b0131e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2526f9aa-9b27-4a09-b321-ec07b6b0131e-maas-api-tls\") pod \"2526f9aa-9b27-4a09-b321-ec07b6b0131e\" (UID: \"2526f9aa-9b27-4a09-b321-ec07b6b0131e\") "
Oct 30 01:22:41.968422 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:41.968107    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2526f9aa-9b27-4a09-b321-ec07b6b0131e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2526f9aa-9b27-4a09-b321-ec07b6b0131e-metrics-certs\") pod \"2526f9aa-9b27-4a09-b321-ec07b6b0131e\" (UID: \"2526f9aa-9b27-4a09-b321-ec07b6b0131e\") "
Oct 30 01:22:41.968422 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:41.968349    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2526f9aa-9b27-4a09-b321-ec07b6b0131e-openshift-service-ca" pod "2526f9aa-9b27-4a09-b321-ec07b6b0131e" (UID: "2526f9aa-9b27-4a09-b321-ec07b6b0131e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:41.969112 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:41.969076    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2526f9aa-9b27-4a09-b321-ec07b6b0131e-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:41.971470 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:41.971441    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2526f9aa-9b27-4a09-b321-ec07b6b0131e-metrics-certs" pod "2526f9aa-9b27-4a09-b321-ec07b6b0131e" (UID: "2526f9aa-9b27-4a09-b321-ec07b6b0131e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:41.971594 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:41.971535    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2526f9aa-9b27-4a09-b321-ec07b6b0131e-maas-api-tls" pod "2526f9aa-9b27-4a09-b321-ec07b6b0131e" (UID: "2526f9aa-9b27-4a09-b321-ec07b6b0131e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:41.971682 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:41.971665    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2526f9aa-9b27-4a09-b321-ec07b6b0131e-kube-api-access-xl9kg" pod "2526f9aa-9b27-4a09-b321-ec07b6b0131e" (UID: "2526f9aa-9b27-4a09-b321-ec07b6b0131e"). InnerVolumeSpecName "kube-api-access-xl9kg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:42.071159 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:42.071122    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl9kg\" (UniqueName: \"kubernetes.io/projected/2526f9aa-9b27-4a09-b321-ec07b6b0131e-kube-api-access-xl9kg\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:42.071159 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:42.071155    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2526f9aa-9b27-4a09-b321-ec07b6b0131e-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:42.071159 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:42.071165    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2526f9aa-9b27-4a09-b321-ec07b6b0131e-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:42.072084 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:42.072047    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462"]
Oct 30 01:22:42.072377 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:42.072350    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462" podUID="38092938-4c3d-41ae-b7dd-25ddd1869d7d" containerName="maas-api" containerID="cri-o://03a5de79d6b901a0cac108e4f045620a918a6760d948fc355f2f0dbe681e2a0d" gracePeriod=30
Oct 30 01:22:42.145837 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:42.145672    4393 generic.go:356] "Generic (PLEG): container finished" podID="2526f9aa-9b27-4a09-b321-ec07b6b0131e" containerID="9363b2c1be68aada576269291ab458aad0361a17baeb10a5494bd9faf6179fd7" exitCode=0
Oct 30 01:22:42.145837 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:42.145764    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm" event={"ID":"2526f9aa-9b27-4a09-b321-ec07b6b0131e","Type":"ContainerDied","Data":"9363b2c1be68aada576269291ab458aad0361a17baeb10a5494bd9faf6179fd7"}
Oct 30 01:22:42.145837 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:42.145837    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm" event={"ID":"2526f9aa-9b27-4a09-b321-ec07b6b0131e","Type":"ContainerDied","Data":"b51b2381a5ba1ecb68f5cb373185236738618e86c5181f418b8e8390bf1546cf"}
Oct 30 01:22:42.151833 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:42.145869    4393 scope.go:122] "RemoveContainer" containerID="9363b2c1be68aada576269291ab458aad0361a17baeb10a5494bd9faf6179fd7"
Oct 30 01:22:42.151833 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:42.146038    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm"
Oct 30 01:22:42.196868 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:42.196838    4393 scope.go:122] "RemoveContainer" containerID="9363b2c1be68aada576269291ab458aad0361a17baeb10a5494bd9faf6179fd7"
Oct 30 01:22:42.198170 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:22:42.198138    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9363b2c1be68aada576269291ab458aad0361a17baeb10a5494bd9faf6179fd7\": container with ID starting with 9363b2c1be68aada576269291ab458aad0361a17baeb10a5494bd9faf6179fd7 not found: ID does not exist" containerID="9363b2c1be68aada576269291ab458aad0361a17baeb10a5494bd9faf6179fd7"
Oct 30 01:22:42.198292 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:42.198183    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9363b2c1be68aada576269291ab458aad0361a17baeb10a5494bd9faf6179fd7"} err="failed to get container status \"9363b2c1be68aada576269291ab458aad0361a17baeb10a5494bd9faf6179fd7\": rpc error: code = NotFound desc = could not find container \"9363b2c1be68aada576269291ab458aad0361a17baeb10a5494bd9faf6179fd7\": container with ID starting with 9363b2c1be68aada576269291ab458aad0361a17baeb10a5494bd9faf6179fd7 not found: ID does not exist"
Oct 30 01:22:42.238596 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:42.238513    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm"]
Oct 30 01:22:42.247535 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:42.247478    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b73232-646cbb5dc4-s59wm"]
Oct 30 01:22:42.346261 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:42.346236    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462"
Oct 30 01:22:42.496565 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:42.496466    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38092938-4c3d-41ae-b7dd-25ddd1869d7d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/38092938-4c3d-41ae-b7dd-25ddd1869d7d-metrics-certs\") pod \"38092938-4c3d-41ae-b7dd-25ddd1869d7d\" (UID: \"38092938-4c3d-41ae-b7dd-25ddd1869d7d\") "
Oct 30 01:22:42.496565 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:42.496531    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38092938-4c3d-41ae-b7dd-25ddd1869d7d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38092938-4c3d-41ae-b7dd-25ddd1869d7d-openshift-service-ca\") pod \"38092938-4c3d-41ae-b7dd-25ddd1869d7d\" (UID: \"38092938-4c3d-41ae-b7dd-25ddd1869d7d\") "
Oct 30 01:22:42.496852 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:42.496658    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38092938-4c3d-41ae-b7dd-25ddd1869d7d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38092938-4c3d-41ae-b7dd-25ddd1869d7d-maas-api-tls\") pod \"38092938-4c3d-41ae-b7dd-25ddd1869d7d\" (UID: \"38092938-4c3d-41ae-b7dd-25ddd1869d7d\") "
Oct 30 01:22:42.496852 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:42.496716    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38092938-4c3d-41ae-b7dd-25ddd1869d7d-kube-api-access-2sqnc\" (UniqueName: \"kubernetes.io/projected/38092938-4c3d-41ae-b7dd-25ddd1869d7d-kube-api-access-2sqnc\") pod \"38092938-4c3d-41ae-b7dd-25ddd1869d7d\" (UID: \"38092938-4c3d-41ae-b7dd-25ddd1869d7d\") "
Oct 30 01:22:42.496975 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:42.496948    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38092938-4c3d-41ae-b7dd-25ddd1869d7d-openshift-service-ca" pod "38092938-4c3d-41ae-b7dd-25ddd1869d7d" (UID: "38092938-4c3d-41ae-b7dd-25ddd1869d7d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:42.497227 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:42.497207    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38092938-4c3d-41ae-b7dd-25ddd1869d7d-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:42.498839 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:42.498816    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38092938-4c3d-41ae-b7dd-25ddd1869d7d-metrics-certs" pod "38092938-4c3d-41ae-b7dd-25ddd1869d7d" (UID: "38092938-4c3d-41ae-b7dd-25ddd1869d7d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:42.499001 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:42.498982    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38092938-4c3d-41ae-b7dd-25ddd1869d7d-maas-api-tls" pod "38092938-4c3d-41ae-b7dd-25ddd1869d7d" (UID: "38092938-4c3d-41ae-b7dd-25ddd1869d7d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:42.499043 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:42.498992    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38092938-4c3d-41ae-b7dd-25ddd1869d7d-kube-api-access-2sqnc" pod "38092938-4c3d-41ae-b7dd-25ddd1869d7d" (UID: "38092938-4c3d-41ae-b7dd-25ddd1869d7d"). InnerVolumeSpecName "kube-api-access-2sqnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:42.598347 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:42.598310    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38092938-4c3d-41ae-b7dd-25ddd1869d7d-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:42.598347 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:42.598343    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38092938-4c3d-41ae-b7dd-25ddd1869d7d-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:42.598347 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:42.598353    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sqnc\" (UniqueName: \"kubernetes.io/projected/38092938-4c3d-41ae-b7dd-25ddd1869d7d-kube-api-access-2sqnc\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:43.154695 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:43.154659    4393 generic.go:356] "Generic (PLEG): container finished" podID="38092938-4c3d-41ae-b7dd-25ddd1869d7d" containerID="03a5de79d6b901a0cac108e4f045620a918a6760d948fc355f2f0dbe681e2a0d" exitCode=0
Oct 30 01:22:43.155170 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:43.154713    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462"
Oct 30 01:22:43.155170 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:43.154748    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462" event={"ID":"38092938-4c3d-41ae-b7dd-25ddd1869d7d","Type":"ContainerDied","Data":"03a5de79d6b901a0cac108e4f045620a918a6760d948fc355f2f0dbe681e2a0d"}
Oct 30 01:22:43.155170 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:43.154838    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462" event={"ID":"38092938-4c3d-41ae-b7dd-25ddd1869d7d","Type":"ContainerDied","Data":"b39d38d7428de526834f1d3d69c2a466f8619f5f26ebd20c8e7dc785171c170a"}
Oct 30 01:22:43.155170 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:43.154870    4393 scope.go:122] "RemoveContainer" containerID="03a5de79d6b901a0cac108e4f045620a918a6760d948fc355f2f0dbe681e2a0d"
Oct 30 01:22:43.175250 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:43.175222    4393 scope.go:122] "RemoveContainer" containerID="03a5de79d6b901a0cac108e4f045620a918a6760d948fc355f2f0dbe681e2a0d"
Oct 30 01:22:43.175600 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:22:43.175575    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03a5de79d6b901a0cac108e4f045620a918a6760d948fc355f2f0dbe681e2a0d\": container with ID starting with 03a5de79d6b901a0cac108e4f045620a918a6760d948fc355f2f0dbe681e2a0d not found: ID does not exist" containerID="03a5de79d6b901a0cac108e4f045620a918a6760d948fc355f2f0dbe681e2a0d"
Oct 30 01:22:43.175700 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:43.175606    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03a5de79d6b901a0cac108e4f045620a918a6760d948fc355f2f0dbe681e2a0d"} err="failed to get container status \"03a5de79d6b901a0cac108e4f045620a918a6760d948fc355f2f0dbe681e2a0d\": rpc error: code = NotFound desc = could not find container \"03a5de79d6b901a0cac108e4f045620a918a6760d948fc355f2f0dbe681e2a0d\": container with ID starting with 03a5de79d6b901a0cac108e4f045620a918a6760d948fc355f2f0dbe681e2a0d not found: ID does not exist"
Oct 30 01:22:43.206642 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:43.206608    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462"]
Oct 30 01:22:43.218275 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:43.218238    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-6f4fba-57fb6d987d-pp462"]
Oct 30 01:22:43.726082 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:43.726038    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2526f9aa-9b27-4a09-b321-ec07b6b0131e" path="/var/lib/kubelet/pods/2526f9aa-9b27-4a09-b321-ec07b6b0131e/volumes"
Oct 30 01:22:43.726588 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:43.726557    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38092938-4c3d-41ae-b7dd-25ddd1869d7d" path="/var/lib/kubelet/pods/38092938-4c3d-41ae-b7dd-25ddd1869d7d/volumes"
Oct 30 01:22:48.265556 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:48.265517    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx"]
Oct 30 01:22:48.266225 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:48.265769    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx" podUID="56a3d613-1b6a-4e50-b5e3-517002e2c5a8" containerName="maas-api" containerID="cri-o://587b916435d155a6010479d1fa562e01ab8ced376a9b0a30512ebc03cf84f082" gracePeriod=30
Oct 30 01:22:48.415454 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:48.415407    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx" podUID="56a3d613-1b6a-4e50-b5e3-517002e2c5a8" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.195:8443/health\": dial tcp 10.133.0.195:8443: connect: connection refused"
Oct 30 01:22:48.555539 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:48.555513    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx"
Oct 30 01:22:48.681030 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:48.680995    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56a3d613-1b6a-4e50-b5e3-517002e2c5a8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56a3d613-1b6a-4e50-b5e3-517002e2c5a8-maas-api-tls\") pod \"56a3d613-1b6a-4e50-b5e3-517002e2c5a8\" (UID: \"56a3d613-1b6a-4e50-b5e3-517002e2c5a8\") "
Oct 30 01:22:48.681250 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:48.681044    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56a3d613-1b6a-4e50-b5e3-517002e2c5a8-kube-api-access-fmkx5\" (UniqueName: \"kubernetes.io/projected/56a3d613-1b6a-4e50-b5e3-517002e2c5a8-kube-api-access-fmkx5\") pod \"56a3d613-1b6a-4e50-b5e3-517002e2c5a8\" (UID: \"56a3d613-1b6a-4e50-b5e3-517002e2c5a8\") "
Oct 30 01:22:48.681250 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:48.681124    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56a3d613-1b6a-4e50-b5e3-517002e2c5a8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a3d613-1b6a-4e50-b5e3-517002e2c5a8-metrics-certs\") pod \"56a3d613-1b6a-4e50-b5e3-517002e2c5a8\" (UID: \"56a3d613-1b6a-4e50-b5e3-517002e2c5a8\") "
Oct 30 01:22:48.681383 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:48.681265    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56a3d613-1b6a-4e50-b5e3-517002e2c5a8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56a3d613-1b6a-4e50-b5e3-517002e2c5a8-openshift-service-ca\") pod \"56a3d613-1b6a-4e50-b5e3-517002e2c5a8\" (UID: \"56a3d613-1b6a-4e50-b5e3-517002e2c5a8\") "
Oct 30 01:22:48.681655 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:48.681619    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56a3d613-1b6a-4e50-b5e3-517002e2c5a8-openshift-service-ca" pod "56a3d613-1b6a-4e50-b5e3-517002e2c5a8" (UID: "56a3d613-1b6a-4e50-b5e3-517002e2c5a8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:22:48.683389 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:48.683363    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56a3d613-1b6a-4e50-b5e3-517002e2c5a8-metrics-certs" pod "56a3d613-1b6a-4e50-b5e3-517002e2c5a8" (UID: "56a3d613-1b6a-4e50-b5e3-517002e2c5a8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:48.683994 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:48.683968    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56a3d613-1b6a-4e50-b5e3-517002e2c5a8-maas-api-tls" pod "56a3d613-1b6a-4e50-b5e3-517002e2c5a8" (UID: "56a3d613-1b6a-4e50-b5e3-517002e2c5a8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:22:48.684091 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:48.684019    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a3d613-1b6a-4e50-b5e3-517002e2c5a8-kube-api-access-fmkx5" pod "56a3d613-1b6a-4e50-b5e3-517002e2c5a8" (UID: "56a3d613-1b6a-4e50-b5e3-517002e2c5a8"). InnerVolumeSpecName "kube-api-access-fmkx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:22:48.783311 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:48.783223    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a3d613-1b6a-4e50-b5e3-517002e2c5a8-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:48.783311 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:48.783255    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56a3d613-1b6a-4e50-b5e3-517002e2c5a8-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:48.783311 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:48.783266    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56a3d613-1b6a-4e50-b5e3-517002e2c5a8-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:48.783311 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:48.783274    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmkx5\" (UniqueName: \"kubernetes.io/projected/56a3d613-1b6a-4e50-b5e3-517002e2c5a8-kube-api-access-fmkx5\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:22:49.199172 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:49.199136    4393 generic.go:356] "Generic (PLEG): container finished" podID="56a3d613-1b6a-4e50-b5e3-517002e2c5a8" containerID="587b916435d155a6010479d1fa562e01ab8ced376a9b0a30512ebc03cf84f082" exitCode=0
Oct 30 01:22:49.199376 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:49.199187    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx"
Oct 30 01:22:49.199376 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:49.199225    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx" event={"ID":"56a3d613-1b6a-4e50-b5e3-517002e2c5a8","Type":"ContainerDied","Data":"587b916435d155a6010479d1fa562e01ab8ced376a9b0a30512ebc03cf84f082"}
Oct 30 01:22:49.199376 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:49.199283    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx" event={"ID":"56a3d613-1b6a-4e50-b5e3-517002e2c5a8","Type":"ContainerDied","Data":"0bf20a73c4b725637330ac85e8da76247ca137398ad928fefc7163953a9913b0"}
Oct 30 01:22:49.199376 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:49.199308    4393 scope.go:122] "RemoveContainer" containerID="587b916435d155a6010479d1fa562e01ab8ced376a9b0a30512ebc03cf84f082"
Oct 30 01:22:49.210713 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:49.210693    4393 scope.go:122] "RemoveContainer" containerID="587b916435d155a6010479d1fa562e01ab8ced376a9b0a30512ebc03cf84f082"
Oct 30 01:22:49.211032 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:22:49.211002    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"587b916435d155a6010479d1fa562e01ab8ced376a9b0a30512ebc03cf84f082\": container with ID starting with 587b916435d155a6010479d1fa562e01ab8ced376a9b0a30512ebc03cf84f082 not found: ID does not exist" containerID="587b916435d155a6010479d1fa562e01ab8ced376a9b0a30512ebc03cf84f082"
Oct 30 01:22:49.211139 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:49.211033    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"587b916435d155a6010479d1fa562e01ab8ced376a9b0a30512ebc03cf84f082"} err="failed to get container status \"587b916435d155a6010479d1fa562e01ab8ced376a9b0a30512ebc03cf84f082\": rpc error: code = NotFound desc = could not find container \"587b916435d155a6010479d1fa562e01ab8ced376a9b0a30512ebc03cf84f082\": container with ID starting with 587b916435d155a6010479d1fa562e01ab8ced376a9b0a30512ebc03cf84f082 not found: ID does not exist"
Oct 30 01:22:49.253400 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:49.253368    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx"]
Oct 30 01:22:49.268472 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:49.267240    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1e8c0e-84dbfc544b-gpfgx"]
Oct 30 01:22:49.725145 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:49.725112    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56a3d613-1b6a-4e50-b5e3-517002e2c5a8" path="/var/lib/kubelet/pods/56a3d613-1b6a-4e50-b5e3-517002e2c5a8/volumes"
Oct 30 01:22:52.375438 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:52.375401    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q"]
Oct 30 01:22:56.514138 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:22:56.514093    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:22:58.005027 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:58.004989    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-cefb27-88db76d58-8gzgh"]
Oct 30 01:22:58.039005 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:22:58.038964    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-cefb27-dfdc89d75-jm8hc"]
Oct 30 01:23:01.745007 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:01.744962    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ca5363-657fbd45cb-wc7w2"]
Oct 30 01:23:01.790447 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:01.790417    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ca5363-57d48d9d99-ctxwr"]
Oct 30 01:23:03.006029 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:03.005978    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-cefb27-88db76d58-8gzgh" podUID="20350585-8ccb-4f19-a634-c2cc4031b330" containerName="payload-processing" containerID="cri-o://d3f57a045e5068597119a4ee91b74a93276b8686798004d37dc39d649d67d205" gracePeriod=25
Oct 30 01:23:03.039680 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:03.039628    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cefb27-dfdc89d75-jm8hc" podUID="691e02fc-7fd6-454b-a78e-31e3c34175c3" containerName="payload-pre-processing" containerID="cri-o://b299afbb36e5c2baf69aed5d23019a145aa46c609984714ea3893de90d4b5a80" gracePeriod=25
Oct 30 01:23:03.186919 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:03.186870    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-cefb27-88db76d58-8gzgh" podUID="20350585-8ccb-4f19-a634-c2cc4031b330" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:23:06.745690 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:06.745644    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-ca5363-657fbd45cb-wc7w2" podUID="dd52771e-289e-4bd7-952e-eaddd165eb58" containerName="payload-processing" containerID="cri-o://8b76f8187d6c3fba98c9febdec57a7b546fe3fb19f56316377a0c2d0bbb3ed87" gracePeriod=25
Oct 30 01:23:06.791006 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:06.790947    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca5363-57d48d9d99-ctxwr" podUID="0fa2bddd-6b15-47a8-95cc-8f952594e284" containerName="payload-pre-processing" containerID="cri-o://89ac78482c6e6fa8c27091a9888db9131e252cc331753f2cf702cd4d5c47e195" gracePeriod=25
Oct 30 01:23:07.397922 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:07.397877    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q" podUID="46b01388-7da7-4e63-95f3-afc811bed013" containerName="main" containerID="cri-o://99c0224efa3f8e76a5e54cd092be370ed6a0a6ef633643fcd7cf2ad75632246a" gracePeriod=45
Oct 30 01:23:07.671366 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:07.671336    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q"
Oct 30 01:23:07.763709 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:07.763672    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b01388-7da7-4e63-95f3-afc811bed013-dshm\" (UniqueName: \"kubernetes.io/empty-dir/46b01388-7da7-4e63-95f3-afc811bed013-dshm\") pod \"46b01388-7da7-4e63-95f3-afc811bed013\" (UID: \"46b01388-7da7-4e63-95f3-afc811bed013\") "
Oct 30 01:23:07.764637 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:07.763722    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b01388-7da7-4e63-95f3-afc811bed013-home\" (UniqueName: \"kubernetes.io/empty-dir/46b01388-7da7-4e63-95f3-afc811bed013-home\") pod \"46b01388-7da7-4e63-95f3-afc811bed013\" (UID: \"46b01388-7da7-4e63-95f3-afc811bed013\") "
Oct 30 01:23:07.764637 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:07.763851    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b01388-7da7-4e63-95f3-afc811bed013-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46b01388-7da7-4e63-95f3-afc811bed013-model-cache\") pod \"46b01388-7da7-4e63-95f3-afc811bed013\" (UID: \"46b01388-7da7-4e63-95f3-afc811bed013\") "
Oct 30 01:23:07.764637 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:07.763897    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46b01388-7da7-4e63-95f3-afc811bed013-tls-certs\" (UniqueName: \"kubernetes.io/secret/46b01388-7da7-4e63-95f3-afc811bed013-tls-certs\") pod \"46b01388-7da7-4e63-95f3-afc811bed013\" (UID: \"46b01388-7da7-4e63-95f3-afc811bed013\") "
Oct 30 01:23:07.764637 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:07.763934    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46b01388-7da7-4e63-95f3-afc811bed013-kube-api-access-mpkcg\" (UniqueName: \"kubernetes.io/projected/46b01388-7da7-4e63-95f3-afc811bed013-kube-api-access-mpkcg\") pod \"46b01388-7da7-4e63-95f3-afc811bed013\" (UID: \"46b01388-7da7-4e63-95f3-afc811bed013\") "
Oct 30 01:23:07.764637 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:07.763983    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b01388-7da7-4e63-95f3-afc811bed013-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46b01388-7da7-4e63-95f3-afc811bed013-tmp-dir\") pod \"46b01388-7da7-4e63-95f3-afc811bed013\" (UID: \"46b01388-7da7-4e63-95f3-afc811bed013\") "
Oct 30 01:23:07.764637 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:07.764523    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b01388-7da7-4e63-95f3-afc811bed013-model-cache" pod "46b01388-7da7-4e63-95f3-afc811bed013" (UID: "46b01388-7da7-4e63-95f3-afc811bed013"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:07.764637 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:07.764620    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b01388-7da7-4e63-95f3-afc811bed013-home" pod "46b01388-7da7-4e63-95f3-afc811bed013" (UID: "46b01388-7da7-4e63-95f3-afc811bed013"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:07.765299 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:07.765270    4393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46b01388-7da7-4e63-95f3-afc811bed013-home\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:07.765299 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:07.765292    4393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46b01388-7da7-4e63-95f3-afc811bed013-model-cache\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:07.765921 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:07.765890    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b01388-7da7-4e63-95f3-afc811bed013-tmp-dir" pod "46b01388-7da7-4e63-95f3-afc811bed013" (UID: "46b01388-7da7-4e63-95f3-afc811bed013"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:07.767146 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:07.767103    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46b01388-7da7-4e63-95f3-afc811bed013-tls-certs" pod "46b01388-7da7-4e63-95f3-afc811bed013" (UID: "46b01388-7da7-4e63-95f3-afc811bed013"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:07.767272 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:07.767191    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b01388-7da7-4e63-95f3-afc811bed013-dshm" pod "46b01388-7da7-4e63-95f3-afc811bed013" (UID: "46b01388-7da7-4e63-95f3-afc811bed013"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:07.767272 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:07.767235    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b01388-7da7-4e63-95f3-afc811bed013-kube-api-access-mpkcg" pod "46b01388-7da7-4e63-95f3-afc811bed013" (UID: "46b01388-7da7-4e63-95f3-afc811bed013"). InnerVolumeSpecName "kube-api-access-mpkcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:07.866656 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:07.866611    4393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46b01388-7da7-4e63-95f3-afc811bed013-dshm\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:07.866656 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:07.866646    4393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46b01388-7da7-4e63-95f3-afc811bed013-tls-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:07.866656 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:07.866657    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpkcg\" (UniqueName: \"kubernetes.io/projected/46b01388-7da7-4e63-95f3-afc811bed013-kube-api-access-mpkcg\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:07.866656 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:07.866665    4393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46b01388-7da7-4e63-95f3-afc811bed013-tmp-dir\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:08.349444 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:08.349405    4393 generic.go:356] "Generic (PLEG): container finished" podID="46b01388-7da7-4e63-95f3-afc811bed013" containerID="99c0224efa3f8e76a5e54cd092be370ed6a0a6ef633643fcd7cf2ad75632246a" exitCode=0
Oct 30 01:23:08.349626 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:08.349452    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q"
Oct 30 01:23:08.349626 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:08.349485    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q" event={"ID":"46b01388-7da7-4e63-95f3-afc811bed013","Type":"ContainerDied","Data":"99c0224efa3f8e76a5e54cd092be370ed6a0a6ef633643fcd7cf2ad75632246a"}
Oct 30 01:23:08.349626 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:08.349530    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q" event={"ID":"46b01388-7da7-4e63-95f3-afc811bed013","Type":"ContainerDied","Data":"d7c571ef5698359dc40d2911163bab63e594662a066fdb86fc63487d03996497"}
Oct 30 01:23:08.349626 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:08.349549    4393 scope.go:122] "RemoveContainer" containerID="99c0224efa3f8e76a5e54cd092be370ed6a0a6ef633643fcd7cf2ad75632246a"
Oct 30 01:23:08.361301 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:08.361281    4393 scope.go:122] "RemoveContainer" containerID="99c0224efa3f8e76a5e54cd092be370ed6a0a6ef633643fcd7cf2ad75632246a"
Oct 30 01:23:08.361666 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:23:08.361642    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99c0224efa3f8e76a5e54cd092be370ed6a0a6ef633643fcd7cf2ad75632246a\": container with ID starting with 99c0224efa3f8e76a5e54cd092be370ed6a0a6ef633643fcd7cf2ad75632246a not found: ID does not exist" containerID="99c0224efa3f8e76a5e54cd092be370ed6a0a6ef633643fcd7cf2ad75632246a"
Oct 30 01:23:08.361743 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:08.361680    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99c0224efa3f8e76a5e54cd092be370ed6a0a6ef633643fcd7cf2ad75632246a"} err="failed to get container status \"99c0224efa3f8e76a5e54cd092be370ed6a0a6ef633643fcd7cf2ad75632246a\": rpc error: code = NotFound desc = could not find container \"99c0224efa3f8e76a5e54cd092be370ed6a0a6ef633643fcd7cf2ad75632246a\": container with ID starting with 99c0224efa3f8e76a5e54cd092be370ed6a0a6ef633643fcd7cf2ad75632246a not found: ID does not exist"
Oct 30 01:23:08.399483 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:08.399440    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q"]
Oct 30 01:23:08.403253 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:08.403222    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-cefb27/distinct-w3-cefb27-kserve-7cbfdf84cc-pm58q"]
Oct 30 01:23:09.724535 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:09.724497    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46b01388-7da7-4e63-95f3-afc811bed013" path="/var/lib/kubelet/pods/46b01388-7da7-4e63-95f3-afc811bed013/volumes"
Oct 30 01:23:10.310828 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:10.310761    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5"]
Oct 30 01:23:10.311797 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:10.311716    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5" podUID="63d6236e-0643-4747-b66d-bbdee696584d" containerName="maas-api" containerID="cri-o://440c7d49ab41e80459df7be5b76f3bfc6d24dd8541d209825f00bef135563570" gracePeriod=30
Oct 30 01:23:10.650376 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:10.650344    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5"
Oct 30 01:23:10.829809 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:10.825197    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63d6236e-0643-4747-b66d-bbdee696584d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63d6236e-0643-4747-b66d-bbdee696584d-maas-api-tls\") pod \"63d6236e-0643-4747-b66d-bbdee696584d\" (UID: \"63d6236e-0643-4747-b66d-bbdee696584d\") "
Oct 30 01:23:10.829809 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:10.825313    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63d6236e-0643-4747-b66d-bbdee696584d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/63d6236e-0643-4747-b66d-bbdee696584d-metrics-certs\") pod \"63d6236e-0643-4747-b66d-bbdee696584d\" (UID: \"63d6236e-0643-4747-b66d-bbdee696584d\") "
Oct 30 01:23:10.829809 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:10.825442    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63d6236e-0643-4747-b66d-bbdee696584d-kube-api-access-68pdj\" (UniqueName: \"kubernetes.io/projected/63d6236e-0643-4747-b66d-bbdee696584d-kube-api-access-68pdj\") pod \"63d6236e-0643-4747-b66d-bbdee696584d\" (UID: \"63d6236e-0643-4747-b66d-bbdee696584d\") "
Oct 30 01:23:10.829809 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:10.825535    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63d6236e-0643-4747-b66d-bbdee696584d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63d6236e-0643-4747-b66d-bbdee696584d-openshift-service-ca\") pod \"63d6236e-0643-4747-b66d-bbdee696584d\" (UID: \"63d6236e-0643-4747-b66d-bbdee696584d\") "
Oct 30 01:23:10.833804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:10.831746    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63d6236e-0643-4747-b66d-bbdee696584d-openshift-service-ca" pod "63d6236e-0643-4747-b66d-bbdee696584d" (UID: "63d6236e-0643-4747-b66d-bbdee696584d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:10.833804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:10.832251    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63d6236e-0643-4747-b66d-bbdee696584d-metrics-certs" pod "63d6236e-0643-4747-b66d-bbdee696584d" (UID: "63d6236e-0643-4747-b66d-bbdee696584d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:10.855387 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:10.855341    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63d6236e-0643-4747-b66d-bbdee696584d-maas-api-tls" pod "63d6236e-0643-4747-b66d-bbdee696584d" (UID: "63d6236e-0643-4747-b66d-bbdee696584d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:10.867131 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:10.867033    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d6236e-0643-4747-b66d-bbdee696584d-kube-api-access-68pdj" pod "63d6236e-0643-4747-b66d-bbdee696584d" (UID: "63d6236e-0643-4747-b66d-bbdee696584d"). InnerVolumeSpecName "kube-api-access-68pdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:10.927934 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:10.927893    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68pdj\" (UniqueName: \"kubernetes.io/projected/63d6236e-0643-4747-b66d-bbdee696584d-kube-api-access-68pdj\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:10.927934 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:10.927937    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63d6236e-0643-4747-b66d-bbdee696584d-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:10.928215 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:10.927949    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63d6236e-0643-4747-b66d-bbdee696584d-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:10.928215 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:10.927957    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63d6236e-0643-4747-b66d-bbdee696584d-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:11.373620 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:11.373577    4393 generic.go:356] "Generic (PLEG): container finished" podID="63d6236e-0643-4747-b66d-bbdee696584d" containerID="440c7d49ab41e80459df7be5b76f3bfc6d24dd8541d209825f00bef135563570" exitCode=0
Oct 30 01:23:11.373877 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:11.373637    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5"
Oct 30 01:23:11.373877 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:11.373664    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5" event={"ID":"63d6236e-0643-4747-b66d-bbdee696584d","Type":"ContainerDied","Data":"440c7d49ab41e80459df7be5b76f3bfc6d24dd8541d209825f00bef135563570"}
Oct 30 01:23:11.373877 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:11.373722    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5" event={"ID":"63d6236e-0643-4747-b66d-bbdee696584d","Type":"ContainerDied","Data":"d77c87bbe8212c141b7f6f5d56b626399976671f72e942655b6eb8aa15d9d49d"}
Oct 30 01:23:11.373877 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:11.373745    4393 scope.go:122] "RemoveContainer" containerID="440c7d49ab41e80459df7be5b76f3bfc6d24dd8541d209825f00bef135563570"
Oct 30 01:23:11.388719 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:11.388652    4393 scope.go:122] "RemoveContainer" containerID="440c7d49ab41e80459df7be5b76f3bfc6d24dd8541d209825f00bef135563570"
Oct 30 01:23:11.389206 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:23:11.389173    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"440c7d49ab41e80459df7be5b76f3bfc6d24dd8541d209825f00bef135563570\": container with ID starting with 440c7d49ab41e80459df7be5b76f3bfc6d24dd8541d209825f00bef135563570 not found: ID does not exist" containerID="440c7d49ab41e80459df7be5b76f3bfc6d24dd8541d209825f00bef135563570"
Oct 30 01:23:11.389295 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:11.389219    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"440c7d49ab41e80459df7be5b76f3bfc6d24dd8541d209825f00bef135563570"} err="failed to get container status \"440c7d49ab41e80459df7be5b76f3bfc6d24dd8541d209825f00bef135563570\": rpc error: code = NotFound desc = could not find container \"440c7d49ab41e80459df7be5b76f3bfc6d24dd8541d209825f00bef135563570\": container with ID starting with 440c7d49ab41e80459df7be5b76f3bfc6d24dd8541d209825f00bef135563570 not found: ID does not exist"
Oct 30 01:23:11.424987 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:11.424930    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5"]
Oct 30 01:23:11.438767 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:11.438723    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-658c99-57d8957b8d-m96f5"]
Oct 30 01:23:11.724808 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:11.724708    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63d6236e-0643-4747-b66d-bbdee696584d" path="/var/lib/kubelet/pods/63d6236e-0643-4747-b66d-bbdee696584d/volumes"
Oct 30 01:23:12.177170 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:12.177004    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cefb27-dfdc89d75-jm8hc" podUID="691e02fc-7fd6-454b-a78e-31e3c34175c3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:23:13.187055 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:13.187015    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-cefb27-88db76d58-8gzgh" podUID="20350585-8ccb-4f19-a634-c2cc4031b330" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:23:14.678811 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:14.678756    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp"]
Oct 30 01:23:14.679359 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:14.679059    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp" podUID="77896a6a-93c0-4f84-91ff-676e591f4453" containerName="maas-api" containerID="cri-o://f5762bc14c89742b92ea2181a18990e77211baadc310e1a43f753f95e80020f0" gracePeriod=30
Oct 30 01:23:14.973611 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:14.973580    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp"
Oct 30 01:23:15.074772 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:15.074731    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77896a6a-93c0-4f84-91ff-676e591f4453-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77896a6a-93c0-4f84-91ff-676e591f4453-maas-api-tls\") pod \"77896a6a-93c0-4f84-91ff-676e591f4453\" (UID: \"77896a6a-93c0-4f84-91ff-676e591f4453\") "
Oct 30 01:23:15.075033 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:15.074847    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77896a6a-93c0-4f84-91ff-676e591f4453-kube-api-access-2j6zd\" (UniqueName: \"kubernetes.io/projected/77896a6a-93c0-4f84-91ff-676e591f4453-kube-api-access-2j6zd\") pod \"77896a6a-93c0-4f84-91ff-676e591f4453\" (UID: \"77896a6a-93c0-4f84-91ff-676e591f4453\") "
Oct 30 01:23:15.077281 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:15.077233    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77896a6a-93c0-4f84-91ff-676e591f4453-maas-api-tls" pod "77896a6a-93c0-4f84-91ff-676e591f4453" (UID: "77896a6a-93c0-4f84-91ff-676e591f4453"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:15.077426 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:15.077306    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77896a6a-93c0-4f84-91ff-676e591f4453-kube-api-access-2j6zd" pod "77896a6a-93c0-4f84-91ff-676e591f4453" (UID: "77896a6a-93c0-4f84-91ff-676e591f4453"). InnerVolumeSpecName "kube-api-access-2j6zd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:15.187840 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:15.178997    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77896a6a-93c0-4f84-91ff-676e591f4453-metrics-certs\" (UniqueName: \"kubernetes.io/secret/77896a6a-93c0-4f84-91ff-676e591f4453-metrics-certs\") pod \"77896a6a-93c0-4f84-91ff-676e591f4453\" (UID: \"77896a6a-93c0-4f84-91ff-676e591f4453\") "
Oct 30 01:23:15.187840 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:15.179062    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77896a6a-93c0-4f84-91ff-676e591f4453-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77896a6a-93c0-4f84-91ff-676e591f4453-openshift-service-ca\") pod \"77896a6a-93c0-4f84-91ff-676e591f4453\" (UID: \"77896a6a-93c0-4f84-91ff-676e591f4453\") "
Oct 30 01:23:15.187840 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:15.179451    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77896a6a-93c0-4f84-91ff-676e591f4453-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:15.187840 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:15.179473    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j6zd\" (UniqueName: \"kubernetes.io/projected/77896a6a-93c0-4f84-91ff-676e591f4453-kube-api-access-2j6zd\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:15.187840 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:15.179882    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77896a6a-93c0-4f84-91ff-676e591f4453-openshift-service-ca" pod "77896a6a-93c0-4f84-91ff-676e591f4453" (UID: "77896a6a-93c0-4f84-91ff-676e591f4453"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:15.187840 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:15.182095    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77896a6a-93c0-4f84-91ff-676e591f4453-metrics-certs" pod "77896a6a-93c0-4f84-91ff-676e591f4453" (UID: "77896a6a-93c0-4f84-91ff-676e591f4453"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:15.208406 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:15.208312    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-ca5363-657fbd45cb-wc7w2" podUID="dd52771e-289e-4bd7-952e-eaddd165eb58" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:23:15.280471 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:15.280429    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77896a6a-93c0-4f84-91ff-676e591f4453-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:15.280471 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:15.280466    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77896a6a-93c0-4f84-91ff-676e591f4453-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:15.419679 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:15.416223    4393 generic.go:356] "Generic (PLEG): container finished" podID="77896a6a-93c0-4f84-91ff-676e591f4453" containerID="f5762bc14c89742b92ea2181a18990e77211baadc310e1a43f753f95e80020f0" exitCode=0
Oct 30 01:23:15.419679 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:15.416389    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp" event={"ID":"77896a6a-93c0-4f84-91ff-676e591f4453","Type":"ContainerDied","Data":"f5762bc14c89742b92ea2181a18990e77211baadc310e1a43f753f95e80020f0"}
Oct 30 01:23:15.419679 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:15.416453    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp" event={"ID":"77896a6a-93c0-4f84-91ff-676e591f4453","Type":"ContainerDied","Data":"4f19f0f1651ea06b4519f80c86017ca619b39839059f7ed9fc66847e9a06e480"}
Oct 30 01:23:15.419679 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:15.416487    4393 scope.go:122] "RemoveContainer" containerID="f5762bc14c89742b92ea2181a18990e77211baadc310e1a43f753f95e80020f0"
Oct 30 01:23:15.419679 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:15.416611    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp"
Oct 30 01:23:15.435438 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:15.435406    4393 scope.go:122] "RemoveContainer" containerID="f5762bc14c89742b92ea2181a18990e77211baadc310e1a43f753f95e80020f0"
Oct 30 01:23:15.435819 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:23:15.435791    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5762bc14c89742b92ea2181a18990e77211baadc310e1a43f753f95e80020f0\": container with ID starting with f5762bc14c89742b92ea2181a18990e77211baadc310e1a43f753f95e80020f0 not found: ID does not exist" containerID="f5762bc14c89742b92ea2181a18990e77211baadc310e1a43f753f95e80020f0"
Oct 30 01:23:15.435943 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:15.435826    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5762bc14c89742b92ea2181a18990e77211baadc310e1a43f753f95e80020f0"} err="failed to get container status \"f5762bc14c89742b92ea2181a18990e77211baadc310e1a43f753f95e80020f0\": rpc error: code = NotFound desc = could not find container \"f5762bc14c89742b92ea2181a18990e77211baadc310e1a43f753f95e80020f0\": container with ID starting with f5762bc14c89742b92ea2181a18990e77211baadc310e1a43f753f95e80020f0 not found: ID does not exist"
Oct 30 01:23:15.472357 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:15.472267    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp"]
Oct 30 01:23:15.476232 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:15.476203    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-cefb27-757799b548-zrvzp"]
Oct 30 01:23:15.725108 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:15.725018    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77896a6a-93c0-4f84-91ff-676e591f4453" path="/var/lib/kubelet/pods/77896a6a-93c0-4f84-91ff-676e591f4453/volumes"
Oct 30 01:23:16.219766 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:16.219724    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca5363-57d48d9d99-ctxwr" podUID="0fa2bddd-6b15-47a8-95cc-8f952594e284" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:23:20.460769 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:20.460730    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht"]
Oct 30 01:23:20.461424 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:20.461005    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht" podUID="4014c968-b2d9-4189-b8fb-18de72da909b" containerName="maas-api" containerID="cri-o://e7f0a1b8006c4e898a13a45b2cd110b6d3c6f975a2fd72281ec9b86da3058362" gracePeriod=30
Oct 30 01:23:20.736306 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:20.736218    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht"
Oct 30 01:23:20.842653 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:20.842607    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4014c968-b2d9-4189-b8fb-18de72da909b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4014c968-b2d9-4189-b8fb-18de72da909b-metrics-certs\") pod \"4014c968-b2d9-4189-b8fb-18de72da909b\" (UID: \"4014c968-b2d9-4189-b8fb-18de72da909b\") "
Oct 30 01:23:20.842917 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:20.842721    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4014c968-b2d9-4189-b8fb-18de72da909b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4014c968-b2d9-4189-b8fb-18de72da909b-maas-api-tls\") pod \"4014c968-b2d9-4189-b8fb-18de72da909b\" (UID: \"4014c968-b2d9-4189-b8fb-18de72da909b\") "
Oct 30 01:23:20.842917 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:20.842771    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4014c968-b2d9-4189-b8fb-18de72da909b-kube-api-access-qh59h\" (UniqueName: \"kubernetes.io/projected/4014c968-b2d9-4189-b8fb-18de72da909b-kube-api-access-qh59h\") pod \"4014c968-b2d9-4189-b8fb-18de72da909b\" (UID: \"4014c968-b2d9-4189-b8fb-18de72da909b\") "
Oct 30 01:23:20.842917 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:20.842862    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4014c968-b2d9-4189-b8fb-18de72da909b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4014c968-b2d9-4189-b8fb-18de72da909b-openshift-service-ca\") pod \"4014c968-b2d9-4189-b8fb-18de72da909b\" (UID: \"4014c968-b2d9-4189-b8fb-18de72da909b\") "
Oct 30 01:23:20.843552 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:20.843518    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4014c968-b2d9-4189-b8fb-18de72da909b-openshift-service-ca" pod "4014c968-b2d9-4189-b8fb-18de72da909b" (UID: "4014c968-b2d9-4189-b8fb-18de72da909b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:20.845386 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:20.845356    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4014c968-b2d9-4189-b8fb-18de72da909b-maas-api-tls" pod "4014c968-b2d9-4189-b8fb-18de72da909b" (UID: "4014c968-b2d9-4189-b8fb-18de72da909b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:20.845503 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:20.845438    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4014c968-b2d9-4189-b8fb-18de72da909b-metrics-certs" pod "4014c968-b2d9-4189-b8fb-18de72da909b" (UID: "4014c968-b2d9-4189-b8fb-18de72da909b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:20.845670 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:20.845637    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4014c968-b2d9-4189-b8fb-18de72da909b-kube-api-access-qh59h" pod "4014c968-b2d9-4189-b8fb-18de72da909b" (UID: "4014c968-b2d9-4189-b8fb-18de72da909b"). InnerVolumeSpecName "kube-api-access-qh59h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:20.955837 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:20.948743    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4014c968-b2d9-4189-b8fb-18de72da909b-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:20.955837 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:20.948802    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4014c968-b2d9-4189-b8fb-18de72da909b-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:20.955837 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:20.948819    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh59h\" (UniqueName: \"kubernetes.io/projected/4014c968-b2d9-4189-b8fb-18de72da909b-kube-api-access-qh59h\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:20.955837 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:20.948834    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4014c968-b2d9-4189-b8fb-18de72da909b-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:21.471310 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:21.471270    4393 generic.go:356] "Generic (PLEG): container finished" podID="4014c968-b2d9-4189-b8fb-18de72da909b" containerID="e7f0a1b8006c4e898a13a45b2cd110b6d3c6f975a2fd72281ec9b86da3058362" exitCode=0
Oct 30 01:23:21.471839 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:21.471315    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht"
Oct 30 01:23:21.471839 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:21.471341    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht" event={"ID":"4014c968-b2d9-4189-b8fb-18de72da909b","Type":"ContainerDied","Data":"e7f0a1b8006c4e898a13a45b2cd110b6d3c6f975a2fd72281ec9b86da3058362"}
Oct 30 01:23:21.471839 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:21.471393    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht" event={"ID":"4014c968-b2d9-4189-b8fb-18de72da909b","Type":"ContainerDied","Data":"600dfa94853eb0a68e2e005d7d602fdcf15a65b313a58c8bbec05edce1c33b1e"}
Oct 30 01:23:21.471839 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:21.471414    4393 scope.go:122] "RemoveContainer" containerID="e7f0a1b8006c4e898a13a45b2cd110b6d3c6f975a2fd72281ec9b86da3058362"
Oct 30 01:23:21.485187 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:21.484903    4393 scope.go:122] "RemoveContainer" containerID="e7f0a1b8006c4e898a13a45b2cd110b6d3c6f975a2fd72281ec9b86da3058362"
Oct 30 01:23:21.485633 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:23:21.485605    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7f0a1b8006c4e898a13a45b2cd110b6d3c6f975a2fd72281ec9b86da3058362\": container with ID starting with e7f0a1b8006c4e898a13a45b2cd110b6d3c6f975a2fd72281ec9b86da3058362 not found: ID does not exist" containerID="e7f0a1b8006c4e898a13a45b2cd110b6d3c6f975a2fd72281ec9b86da3058362"
Oct 30 01:23:21.485718 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:21.485648    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7f0a1b8006c4e898a13a45b2cd110b6d3c6f975a2fd72281ec9b86da3058362"} err="failed to get container status \"e7f0a1b8006c4e898a13a45b2cd110b6d3c6f975a2fd72281ec9b86da3058362\": rpc error: code = NotFound desc = could not find container \"e7f0a1b8006c4e898a13a45b2cd110b6d3c6f975a2fd72281ec9b86da3058362\": container with ID starting with e7f0a1b8006c4e898a13a45b2cd110b6d3c6f975a2fd72281ec9b86da3058362 not found: ID does not exist"
Oct 30 01:23:21.526234 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:21.526196    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht"]
Oct 30 01:23:21.537840 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:21.537810    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f64dc0-5b6d67b45b-j9nht"]
Oct 30 01:23:21.725180 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:21.725097    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4014c968-b2d9-4189-b8fb-18de72da909b" path="/var/lib/kubelet/pods/4014c968-b2d9-4189-b8fb-18de72da909b/volumes"
Oct 30 01:23:22.195582 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:22.195538    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cefb27-dfdc89d75-jm8hc" podUID="691e02fc-7fd6-454b-a78e-31e3c34175c3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:23:23.088826 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.088793    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq"]
Oct 30 01:23:23.089279 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.089081    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq" podUID="fb10b912-9abe-4547-83a8-aa63ddb14599" containerName="maas-api" containerID="cri-o://d0daf3efb359bcf377aa5b92bad2d929210896d7582a1c9b11dfc1d8b0df075d" gracePeriod=30
Oct 30 01:23:23.186501 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:23:23.186467    4393 logging.go:55] [core] [Channel #10951 SubChannel #10952]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.57:50052", ServerName: "10.133.0.57:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.57:50052: connect: connection refused"
Oct 30 01:23:23.412002 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.411980    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-cefb27-88db76d58-8gzgh"
Oct 30 01:23:23.424111 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.420957    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cefb27-dfdc89d75-jm8hc"
Oct 30 01:23:23.474677 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.474644    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq"
Oct 30 01:23:23.482209 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.482181    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20350585-8ccb-4f19-a634-c2cc4031b330-kube-api-access-z8x55\" (UniqueName: \"kubernetes.io/projected/20350585-8ccb-4f19-a634-c2cc4031b330-kube-api-access-z8x55\") pod \"20350585-8ccb-4f19-a634-c2cc4031b330\" (UID: \"20350585-8ccb-4f19-a634-c2cc4031b330\") "
Oct 30 01:23:23.482376 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.482249    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/691e02fc-7fd6-454b-a78e-31e3c34175c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/691e02fc-7fd6-454b-a78e-31e3c34175c3-plugins-config-volume\") pod \"691e02fc-7fd6-454b-a78e-31e3c34175c3\" (UID: \"691e02fc-7fd6-454b-a78e-31e3c34175c3\") "
Oct 30 01:23:23.482376 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.482351    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/691e02fc-7fd6-454b-a78e-31e3c34175c3-kube-api-access-h2hlb\" (UniqueName: \"kubernetes.io/projected/691e02fc-7fd6-454b-a78e-31e3c34175c3-kube-api-access-h2hlb\") pod \"691e02fc-7fd6-454b-a78e-31e3c34175c3\" (UID: \"691e02fc-7fd6-454b-a78e-31e3c34175c3\") "
Oct 30 01:23:23.482723 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.482436    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20350585-8ccb-4f19-a634-c2cc4031b330-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20350585-8ccb-4f19-a634-c2cc4031b330-plugins-config-volume\") pod \"20350585-8ccb-4f19-a634-c2cc4031b330\" (UID: \"20350585-8ccb-4f19-a634-c2cc4031b330\") "
Oct 30 01:23:23.482723 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.482678    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/691e02fc-7fd6-454b-a78e-31e3c34175c3-plugins-config-volume" pod "691e02fc-7fd6-454b-a78e-31e3c34175c3" (UID: "691e02fc-7fd6-454b-a78e-31e3c34175c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:23.483039 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.483016    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/691e02fc-7fd6-454b-a78e-31e3c34175c3-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:23.483155 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.483106    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20350585-8ccb-4f19-a634-c2cc4031b330-plugins-config-volume" pod "20350585-8ccb-4f19-a634-c2cc4031b330" (UID: "20350585-8ccb-4f19-a634-c2cc4031b330"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:23.485047 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.485022    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20350585-8ccb-4f19-a634-c2cc4031b330-kube-api-access-z8x55" pod "20350585-8ccb-4f19-a634-c2cc4031b330" (UID: "20350585-8ccb-4f19-a634-c2cc4031b330"). InnerVolumeSpecName "kube-api-access-z8x55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:23.485273 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.485246    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/691e02fc-7fd6-454b-a78e-31e3c34175c3-kube-api-access-h2hlb" pod "691e02fc-7fd6-454b-a78e-31e3c34175c3" (UID: "691e02fc-7fd6-454b-a78e-31e3c34175c3"). InnerVolumeSpecName "kube-api-access-h2hlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:23.492108 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.492081    4393 generic.go:356] "Generic (PLEG): container finished" podID="691e02fc-7fd6-454b-a78e-31e3c34175c3" containerID="b299afbb36e5c2baf69aed5d23019a145aa46c609984714ea3893de90d4b5a80" exitCode=0
Oct 30 01:23:23.492305 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.492276    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cefb27-dfdc89d75-jm8hc" event={"ID":"691e02fc-7fd6-454b-a78e-31e3c34175c3","Type":"ContainerDied","Data":"b299afbb36e5c2baf69aed5d23019a145aa46c609984714ea3893de90d4b5a80"}
Oct 30 01:23:23.492495 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.492443    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cefb27-dfdc89d75-jm8hc" event={"ID":"691e02fc-7fd6-454b-a78e-31e3c34175c3","Type":"ContainerDied","Data":"bcb7c68e5438240d2dabe844e2502ccfcb768349e409489bda89f881492a3301"}
Oct 30 01:23:23.492495 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.492479    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-cefb27-dfdc89d75-jm8hc"
Oct 30 01:23:23.493406 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.492575    4393 scope.go:122] "RemoveContainer" containerID="b299afbb36e5c2baf69aed5d23019a145aa46c609984714ea3893de90d4b5a80"
Oct 30 01:23:23.498096 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.496272    4393 generic.go:356] "Generic (PLEG): container finished" podID="20350585-8ccb-4f19-a634-c2cc4031b330" containerID="d3f57a045e5068597119a4ee91b74a93276b8686798004d37dc39d649d67d205" exitCode=0
Oct 30 01:23:23.498096 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.496456    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-cefb27-88db76d58-8gzgh" event={"ID":"20350585-8ccb-4f19-a634-c2cc4031b330","Type":"ContainerDied","Data":"d3f57a045e5068597119a4ee91b74a93276b8686798004d37dc39d649d67d205"}
Oct 30 01:23:23.498096 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.496516    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-cefb27-88db76d58-8gzgh" event={"ID":"20350585-8ccb-4f19-a634-c2cc4031b330","Type":"ContainerDied","Data":"f84fbc7066ad275a6c8a5cf6b3977c13d81f6d1a144c4cff3ab6387e05143a70"}
Oct 30 01:23:23.498096 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.496596    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-cefb27-88db76d58-8gzgh"
Oct 30 01:23:23.523073 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.512682    4393 generic.go:356] "Generic (PLEG): container finished" podID="fb10b912-9abe-4547-83a8-aa63ddb14599" containerID="d0daf3efb359bcf377aa5b92bad2d929210896d7582a1c9b11dfc1d8b0df075d" exitCode=0
Oct 30 01:23:23.523073 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.512722    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq" event={"ID":"fb10b912-9abe-4547-83a8-aa63ddb14599","Type":"ContainerDied","Data":"d0daf3efb359bcf377aa5b92bad2d929210896d7582a1c9b11dfc1d8b0df075d"}
Oct 30 01:23:23.523073 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.512769    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq" event={"ID":"fb10b912-9abe-4547-83a8-aa63ddb14599","Type":"ContainerDied","Data":"f17d85b70fb27acbd70372669a6ba00eef25c8c27d81310aa048cdd5ef6a8896"}
Oct 30 01:23:23.523073 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.512875    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq"
Oct 30 01:23:23.584495 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.584460    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb10b912-9abe-4547-83a8-aa63ddb14599-kube-api-access-5wlwc\" (UniqueName: \"kubernetes.io/projected/fb10b912-9abe-4547-83a8-aa63ddb14599-kube-api-access-5wlwc\") pod \"fb10b912-9abe-4547-83a8-aa63ddb14599\" (UID: \"fb10b912-9abe-4547-83a8-aa63ddb14599\") "
Oct 30 01:23:23.584681 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.584595    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb10b912-9abe-4547-83a8-aa63ddb14599-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb10b912-9abe-4547-83a8-aa63ddb14599-openshift-service-ca\") pod \"fb10b912-9abe-4547-83a8-aa63ddb14599\" (UID: \"fb10b912-9abe-4547-83a8-aa63ddb14599\") "
Oct 30 01:23:23.584768 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.584688    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb10b912-9abe-4547-83a8-aa63ddb14599-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb10b912-9abe-4547-83a8-aa63ddb14599-maas-api-tls\") pod \"fb10b912-9abe-4547-83a8-aa63ddb14599\" (UID: \"fb10b912-9abe-4547-83a8-aa63ddb14599\") "
Oct 30 01:23:23.584768 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.584739    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb10b912-9abe-4547-83a8-aa63ddb14599-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb10b912-9abe-4547-83a8-aa63ddb14599-metrics-certs\") pod \"fb10b912-9abe-4547-83a8-aa63ddb14599\" (UID: \"fb10b912-9abe-4547-83a8-aa63ddb14599\") "
Oct 30 01:23:23.585500 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.585470    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20350585-8ccb-4f19-a634-c2cc4031b330-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:23.585641 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.585503    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8x55\" (UniqueName: \"kubernetes.io/projected/20350585-8ccb-4f19-a634-c2cc4031b330-kube-api-access-z8x55\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:23.585641 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.585518    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2hlb\" (UniqueName: \"kubernetes.io/projected/691e02fc-7fd6-454b-a78e-31e3c34175c3-kube-api-access-h2hlb\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:23.586284 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.586246    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb10b912-9abe-4547-83a8-aa63ddb14599-openshift-service-ca" pod "fb10b912-9abe-4547-83a8-aa63ddb14599" (UID: "fb10b912-9abe-4547-83a8-aa63ddb14599"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:23.588957 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.588923    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb10b912-9abe-4547-83a8-aa63ddb14599-maas-api-tls" pod "fb10b912-9abe-4547-83a8-aa63ddb14599" (UID: "fb10b912-9abe-4547-83a8-aa63ddb14599"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:23.589093 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.589073    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb10b912-9abe-4547-83a8-aa63ddb14599-kube-api-access-5wlwc" pod "fb10b912-9abe-4547-83a8-aa63ddb14599" (UID: "fb10b912-9abe-4547-83a8-aa63ddb14599"). InnerVolumeSpecName "kube-api-access-5wlwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:23.590032 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.589811    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb10b912-9abe-4547-83a8-aa63ddb14599-metrics-certs" pod "fb10b912-9abe-4547-83a8-aa63ddb14599" (UID: "fb10b912-9abe-4547-83a8-aa63ddb14599"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:23.593939 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.593914    4393 scope.go:122] "RemoveContainer" containerID="b299afbb36e5c2baf69aed5d23019a145aa46c609984714ea3893de90d4b5a80"
Oct 30 01:23:23.594322 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:23:23.594295    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b299afbb36e5c2baf69aed5d23019a145aa46c609984714ea3893de90d4b5a80\": container with ID starting with b299afbb36e5c2baf69aed5d23019a145aa46c609984714ea3893de90d4b5a80 not found: ID does not exist" containerID="b299afbb36e5c2baf69aed5d23019a145aa46c609984714ea3893de90d4b5a80"
Oct 30 01:23:23.594414 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.594330    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b299afbb36e5c2baf69aed5d23019a145aa46c609984714ea3893de90d4b5a80"} err="failed to get container status \"b299afbb36e5c2baf69aed5d23019a145aa46c609984714ea3893de90d4b5a80\": rpc error: code = NotFound desc = could not find container \"b299afbb36e5c2baf69aed5d23019a145aa46c609984714ea3893de90d4b5a80\": container with ID starting with b299afbb36e5c2baf69aed5d23019a145aa46c609984714ea3893de90d4b5a80 not found: ID does not exist"
Oct 30 01:23:23.594414 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.594349    4393 scope.go:122] "RemoveContainer" containerID="d3f57a045e5068597119a4ee91b74a93276b8686798004d37dc39d649d67d205"
Oct 30 01:23:23.605346 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.605311    4393 scope.go:122] "RemoveContainer" containerID="d3f57a045e5068597119a4ee91b74a93276b8686798004d37dc39d649d67d205"
Oct 30 01:23:23.605827 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:23:23.605794    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3f57a045e5068597119a4ee91b74a93276b8686798004d37dc39d649d67d205\": container with ID starting with d3f57a045e5068597119a4ee91b74a93276b8686798004d37dc39d649d67d205 not found: ID does not exist" containerID="d3f57a045e5068597119a4ee91b74a93276b8686798004d37dc39d649d67d205"
Oct 30 01:23:23.605925 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.605846    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3f57a045e5068597119a4ee91b74a93276b8686798004d37dc39d649d67d205"} err="failed to get container status \"d3f57a045e5068597119a4ee91b74a93276b8686798004d37dc39d649d67d205\": rpc error: code = NotFound desc = could not find container \"d3f57a045e5068597119a4ee91b74a93276b8686798004d37dc39d649d67d205\": container with ID starting with d3f57a045e5068597119a4ee91b74a93276b8686798004d37dc39d649d67d205 not found: ID does not exist"
Oct 30 01:23:23.605925 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.605869    4393 scope.go:122] "RemoveContainer" containerID="d0daf3efb359bcf377aa5b92bad2d929210896d7582a1c9b11dfc1d8b0df075d"
Oct 30 01:23:23.615927 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.615887    4393 scope.go:122] "RemoveContainer" containerID="d0daf3efb359bcf377aa5b92bad2d929210896d7582a1c9b11dfc1d8b0df075d"
Oct 30 01:23:23.616240 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:23:23.616218    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0daf3efb359bcf377aa5b92bad2d929210896d7582a1c9b11dfc1d8b0df075d\": container with ID starting with d0daf3efb359bcf377aa5b92bad2d929210896d7582a1c9b11dfc1d8b0df075d not found: ID does not exist" containerID="d0daf3efb359bcf377aa5b92bad2d929210896d7582a1c9b11dfc1d8b0df075d"
Oct 30 01:23:23.616300 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.616250    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0daf3efb359bcf377aa5b92bad2d929210896d7582a1c9b11dfc1d8b0df075d"} err="failed to get container status \"d0daf3efb359bcf377aa5b92bad2d929210896d7582a1c9b11dfc1d8b0df075d\": rpc error: code = NotFound desc = could not find container \"d0daf3efb359bcf377aa5b92bad2d929210896d7582a1c9b11dfc1d8b0df075d\": container with ID starting with d0daf3efb359bcf377aa5b92bad2d929210896d7582a1c9b11dfc1d8b0df075d not found: ID does not exist"
Oct 30 01:23:23.650208 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.650171    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-cefb27-dfdc89d75-jm8hc"]
Oct 30 01:23:23.654642 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.654612    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-cefb27-dfdc89d75-jm8hc"]
Oct 30 01:23:23.686453 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.686418    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb10b912-9abe-4547-83a8-aa63ddb14599-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:23.686453 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.686456    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb10b912-9abe-4547-83a8-aa63ddb14599-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:23.686672 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.686471    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb10b912-9abe-4547-83a8-aa63ddb14599-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:23.686672 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.686484    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wlwc\" (UniqueName: \"kubernetes.io/projected/fb10b912-9abe-4547-83a8-aa63ddb14599-kube-api-access-5wlwc\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:23.687374 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.687348    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-cefb27-88db76d58-8gzgh"]
Oct 30 01:23:23.690769 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.690749    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-cefb27-88db76d58-8gzgh"]
Oct 30 01:23:23.723982 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.723955    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20350585-8ccb-4f19-a634-c2cc4031b330" path="/var/lib/kubelet/pods/20350585-8ccb-4f19-a634-c2cc4031b330/volumes"
Oct 30 01:23:23.724373 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.724360    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="691e02fc-7fd6-454b-a78e-31e3c34175c3" path="/var/lib/kubelet/pods/691e02fc-7fd6-454b-a78e-31e3c34175c3/volumes"
Oct 30 01:23:23.853866 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.853752    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq"]
Oct 30 01:23:23.856701 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:23.856672    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ca5363-79dc98674c-vbzbq"]
Oct 30 01:23:24.186881 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:24.186842    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-cefb27-88db76d58-8gzgh" podUID="20350585-8ccb-4f19-a634-c2cc4031b330" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.57:50052\" within 1s: context deadline exceeded"
Oct 30 01:23:25.207963 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:25.207927    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-ca5363-657fbd45cb-wc7w2" podUID="dd52771e-289e-4bd7-952e-eaddd165eb58" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:23:25.724793 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:25.724746    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb10b912-9abe-4547-83a8-aa63ddb14599" path="/var/lib/kubelet/pods/fb10b912-9abe-4547-83a8-aa63ddb14599/volumes"
Oct 30 01:23:26.219173 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:26.219133    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca5363-57d48d9d99-ctxwr" podUID="0fa2bddd-6b15-47a8-95cc-8f952594e284" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:23:26.511124 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:23:26.511033    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:23:27.051886 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:27.051854    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ca5363-657fbd45cb-wc7w2"
Oct 30 01:23:27.096210 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:27.096181    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca5363-57d48d9d99-ctxwr"
Oct 30 01:23:27.130952 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:27.130899    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd52771e-289e-4bd7-952e-eaddd165eb58-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd52771e-289e-4bd7-952e-eaddd165eb58-plugins-config-volume\") pod \"dd52771e-289e-4bd7-952e-eaddd165eb58\" (UID: \"dd52771e-289e-4bd7-952e-eaddd165eb58\") "
Oct 30 01:23:27.131162 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:27.130963    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd52771e-289e-4bd7-952e-eaddd165eb58-kube-api-access-rngzf\" (UniqueName: \"kubernetes.io/projected/dd52771e-289e-4bd7-952e-eaddd165eb58-kube-api-access-rngzf\") pod \"dd52771e-289e-4bd7-952e-eaddd165eb58\" (UID: \"dd52771e-289e-4bd7-952e-eaddd165eb58\") "
Oct 30 01:23:27.131402 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:27.131376    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd52771e-289e-4bd7-952e-eaddd165eb58-plugins-config-volume" pod "dd52771e-289e-4bd7-952e-eaddd165eb58" (UID: "dd52771e-289e-4bd7-952e-eaddd165eb58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:27.131582 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:27.131561    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd52771e-289e-4bd7-952e-eaddd165eb58-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:27.133482 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:27.133446    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd52771e-289e-4bd7-952e-eaddd165eb58-kube-api-access-rngzf" pod "dd52771e-289e-4bd7-952e-eaddd165eb58" (UID: "dd52771e-289e-4bd7-952e-eaddd165eb58"). InnerVolumeSpecName "kube-api-access-rngzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:27.232926 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:27.232887    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fa2bddd-6b15-47a8-95cc-8f952594e284-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa2bddd-6b15-47a8-95cc-8f952594e284-plugins-config-volume\") pod \"0fa2bddd-6b15-47a8-95cc-8f952594e284\" (UID: \"0fa2bddd-6b15-47a8-95cc-8f952594e284\") "
Oct 30 01:23:27.233324 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:27.233023    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fa2bddd-6b15-47a8-95cc-8f952594e284-kube-api-access-z7zwp\" (UniqueName: \"kubernetes.io/projected/0fa2bddd-6b15-47a8-95cc-8f952594e284-kube-api-access-z7zwp\") pod \"0fa2bddd-6b15-47a8-95cc-8f952594e284\" (UID: \"0fa2bddd-6b15-47a8-95cc-8f952594e284\") "
Oct 30 01:23:27.233324 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:27.233252    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fa2bddd-6b15-47a8-95cc-8f952594e284-plugins-config-volume" pod "0fa2bddd-6b15-47a8-95cc-8f952594e284" (UID: "0fa2bddd-6b15-47a8-95cc-8f952594e284"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:27.233400 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:27.233377    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rngzf\" (UniqueName: \"kubernetes.io/projected/dd52771e-289e-4bd7-952e-eaddd165eb58-kube-api-access-rngzf\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:27.233400 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:27.233390    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa2bddd-6b15-47a8-95cc-8f952594e284-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:27.235291 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:27.235274    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fa2bddd-6b15-47a8-95cc-8f952594e284-kube-api-access-z7zwp" pod "0fa2bddd-6b15-47a8-95cc-8f952594e284" (UID: "0fa2bddd-6b15-47a8-95cc-8f952594e284"). InnerVolumeSpecName "kube-api-access-z7zwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:27.334297 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:27.334205    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7zwp\" (UniqueName: \"kubernetes.io/projected/0fa2bddd-6b15-47a8-95cc-8f952594e284-kube-api-access-z7zwp\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:27.544642 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:27.544604    4393 generic.go:356] "Generic (PLEG): container finished" podID="dd52771e-289e-4bd7-952e-eaddd165eb58" containerID="8b76f8187d6c3fba98c9febdec57a7b546fe3fb19f56316377a0c2d0bbb3ed87" exitCode=0
Oct 30 01:23:27.544871 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:27.544649    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ca5363-657fbd45cb-wc7w2"
Oct 30 01:23:27.544871 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:27.544690    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ca5363-657fbd45cb-wc7w2" event={"ID":"dd52771e-289e-4bd7-952e-eaddd165eb58","Type":"ContainerDied","Data":"8b76f8187d6c3fba98c9febdec57a7b546fe3fb19f56316377a0c2d0bbb3ed87"}
Oct 30 01:23:27.544871 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:27.544747    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ca5363-657fbd45cb-wc7w2" event={"ID":"dd52771e-289e-4bd7-952e-eaddd165eb58","Type":"ContainerDied","Data":"520e946f022b988a79a61dfcb0edceb643e7c6143ff535a8c25dfb296e450cb5"}
Oct 30 01:23:27.544871 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:27.544763    4393 scope.go:122] "RemoveContainer" containerID="8b76f8187d6c3fba98c9febdec57a7b546fe3fb19f56316377a0c2d0bbb3ed87"
Oct 30 01:23:27.546331 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:27.546305    4393 generic.go:356] "Generic (PLEG): container finished" podID="0fa2bddd-6b15-47a8-95cc-8f952594e284" containerID="89ac78482c6e6fa8c27091a9888db9131e252cc331753f2cf702cd4d5c47e195" exitCode=0
Oct 30 01:23:27.546492 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:27.546366    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca5363-57d48d9d99-ctxwr"
Oct 30 01:23:27.546492 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:27.546432    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca5363-57d48d9d99-ctxwr" event={"ID":"0fa2bddd-6b15-47a8-95cc-8f952594e284","Type":"ContainerDied","Data":"89ac78482c6e6fa8c27091a9888db9131e252cc331753f2cf702cd4d5c47e195"}
Oct 30 01:23:27.546612 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:27.546490    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca5363-57d48d9d99-ctxwr" event={"ID":"0fa2bddd-6b15-47a8-95cc-8f952594e284","Type":"ContainerDied","Data":"0b2ed3b08df987477a5538f8c4195463b26622429716dba9ab5b3226ffec3752"}
Oct 30 01:23:27.555139 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:27.555121    4393 scope.go:122] "RemoveContainer" containerID="8b76f8187d6c3fba98c9febdec57a7b546fe3fb19f56316377a0c2d0bbb3ed87"
Oct 30 01:23:27.555445 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:23:27.555417    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b76f8187d6c3fba98c9febdec57a7b546fe3fb19f56316377a0c2d0bbb3ed87\": container with ID starting with 8b76f8187d6c3fba98c9febdec57a7b546fe3fb19f56316377a0c2d0bbb3ed87 not found: ID does not exist" containerID="8b76f8187d6c3fba98c9febdec57a7b546fe3fb19f56316377a0c2d0bbb3ed87"
Oct 30 01:23:27.555529 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:27.555458    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b76f8187d6c3fba98c9febdec57a7b546fe3fb19f56316377a0c2d0bbb3ed87"} err="failed to get container status \"8b76f8187d6c3fba98c9febdec57a7b546fe3fb19f56316377a0c2d0bbb3ed87\": rpc error: code = NotFound desc = could not find container \"8b76f8187d6c3fba98c9febdec57a7b546fe3fb19f56316377a0c2d0bbb3ed87\": container with ID starting with 8b76f8187d6c3fba98c9febdec57a7b546fe3fb19f56316377a0c2d0bbb3ed87 not found: ID does not exist"
Oct 30 01:23:27.555529 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:27.555482    4393 scope.go:122] "RemoveContainer" containerID="89ac78482c6e6fa8c27091a9888db9131e252cc331753f2cf702cd4d5c47e195"
Oct 30 01:23:27.564629 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:27.564613    4393 scope.go:122] "RemoveContainer" containerID="89ac78482c6e6fa8c27091a9888db9131e252cc331753f2cf702cd4d5c47e195"
Oct 30 01:23:27.564947 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:23:27.564928    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89ac78482c6e6fa8c27091a9888db9131e252cc331753f2cf702cd4d5c47e195\": container with ID starting with 89ac78482c6e6fa8c27091a9888db9131e252cc331753f2cf702cd4d5c47e195 not found: ID does not exist" containerID="89ac78482c6e6fa8c27091a9888db9131e252cc331753f2cf702cd4d5c47e195"
Oct 30 01:23:27.565003 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:27.564956    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89ac78482c6e6fa8c27091a9888db9131e252cc331753f2cf702cd4d5c47e195"} err="failed to get container status \"89ac78482c6e6fa8c27091a9888db9131e252cc331753f2cf702cd4d5c47e195\": rpc error: code = NotFound desc = could not find container \"89ac78482c6e6fa8c27091a9888db9131e252cc331753f2cf702cd4d5c47e195\": container with ID starting with 89ac78482c6e6fa8c27091a9888db9131e252cc331753f2cf702cd4d5c47e195 not found: ID does not exist"
Oct 30 01:23:27.602950 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:27.598718    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ca5363-657fbd45cb-wc7w2"]
Oct 30 01:23:27.611885 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:27.611853    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ca5363-657fbd45cb-wc7w2"]
Oct 30 01:23:27.639234 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:27.639207    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ca5363-57d48d9d99-ctxwr"]
Oct 30 01:23:27.641937 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:27.641914    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ca5363-57d48d9d99-ctxwr"]
Oct 30 01:23:27.724350 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:27.724315    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fa2bddd-6b15-47a8-95cc-8f952594e284" path="/var/lib/kubelet/pods/0fa2bddd-6b15-47a8-95cc-8f952594e284/volumes"
Oct 30 01:23:27.724729 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:27.724715    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd52771e-289e-4bd7-952e-eaddd165eb58" path="/var/lib/kubelet/pods/dd52771e-289e-4bd7-952e-eaddd165eb58/volumes"
Oct 30 01:23:30.854481 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:30.854390    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md"]
Oct 30 01:23:30.854977 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:30.854643    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md" podUID="7028bbc0-dd7e-4525-a1db-8bf264d8ad58" containerName="maas-api" containerID="cri-o://ea7a05c5bd5662b0a96f950b902237681c33f95c0066beb4a4eec5e717690dd0" gracePeriod=30
Oct 30 01:23:31.135536 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:31.135508    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md"
Oct 30 01:23:31.305974 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:31.305934    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7028bbc0-dd7e-4525-a1db-8bf264d8ad58-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7028bbc0-dd7e-4525-a1db-8bf264d8ad58-maas-api-tls\") pod \"7028bbc0-dd7e-4525-a1db-8bf264d8ad58\" (UID: \"7028bbc0-dd7e-4525-a1db-8bf264d8ad58\") "
Oct 30 01:23:31.306167 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:31.306052    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7028bbc0-dd7e-4525-a1db-8bf264d8ad58-kube-api-access-chx5r\" (UniqueName: \"kubernetes.io/projected/7028bbc0-dd7e-4525-a1db-8bf264d8ad58-kube-api-access-chx5r\") pod \"7028bbc0-dd7e-4525-a1db-8bf264d8ad58\" (UID: \"7028bbc0-dd7e-4525-a1db-8bf264d8ad58\") "
Oct 30 01:23:31.306167 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:31.306123    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7028bbc0-dd7e-4525-a1db-8bf264d8ad58-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7028bbc0-dd7e-4525-a1db-8bf264d8ad58-metrics-certs\") pod \"7028bbc0-dd7e-4525-a1db-8bf264d8ad58\" (UID: \"7028bbc0-dd7e-4525-a1db-8bf264d8ad58\") "
Oct 30 01:23:31.306250 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:31.306208    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7028bbc0-dd7e-4525-a1db-8bf264d8ad58-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7028bbc0-dd7e-4525-a1db-8bf264d8ad58-openshift-service-ca\") pod \"7028bbc0-dd7e-4525-a1db-8bf264d8ad58\" (UID: \"7028bbc0-dd7e-4525-a1db-8bf264d8ad58\") "
Oct 30 01:23:31.306652 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:31.306619    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7028bbc0-dd7e-4525-a1db-8bf264d8ad58-openshift-service-ca" pod "7028bbc0-dd7e-4525-a1db-8bf264d8ad58" (UID: "7028bbc0-dd7e-4525-a1db-8bf264d8ad58"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:31.308358 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:31.308333    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7028bbc0-dd7e-4525-a1db-8bf264d8ad58-metrics-certs" pod "7028bbc0-dd7e-4525-a1db-8bf264d8ad58" (UID: "7028bbc0-dd7e-4525-a1db-8bf264d8ad58"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:31.308451 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:31.308359    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7028bbc0-dd7e-4525-a1db-8bf264d8ad58-maas-api-tls" pod "7028bbc0-dd7e-4525-a1db-8bf264d8ad58" (UID: "7028bbc0-dd7e-4525-a1db-8bf264d8ad58"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:31.308492 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:31.308447    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7028bbc0-dd7e-4525-a1db-8bf264d8ad58-kube-api-access-chx5r" pod "7028bbc0-dd7e-4525-a1db-8bf264d8ad58" (UID: "7028bbc0-dd7e-4525-a1db-8bf264d8ad58"). InnerVolumeSpecName "kube-api-access-chx5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:31.424857 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:31.418620    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chx5r\" (UniqueName: \"kubernetes.io/projected/7028bbc0-dd7e-4525-a1db-8bf264d8ad58-kube-api-access-chx5r\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:31.424857 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:31.418659    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7028bbc0-dd7e-4525-a1db-8bf264d8ad58-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:31.424857 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:31.418678    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7028bbc0-dd7e-4525-a1db-8bf264d8ad58-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:31.424857 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:31.418692    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7028bbc0-dd7e-4525-a1db-8bf264d8ad58-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:31.603345 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:31.603308    4393 generic.go:356] "Generic (PLEG): container finished" podID="7028bbc0-dd7e-4525-a1db-8bf264d8ad58" containerID="ea7a05c5bd5662b0a96f950b902237681c33f95c0066beb4a4eec5e717690dd0" exitCode=0
Oct 30 01:23:31.603526 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:31.603358    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md"
Oct 30 01:23:31.603526 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:31.603385    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md" event={"ID":"7028bbc0-dd7e-4525-a1db-8bf264d8ad58","Type":"ContainerDied","Data":"ea7a05c5bd5662b0a96f950b902237681c33f95c0066beb4a4eec5e717690dd0"}
Oct 30 01:23:31.603526 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:31.603444    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md" event={"ID":"7028bbc0-dd7e-4525-a1db-8bf264d8ad58","Type":"ContainerDied","Data":"64eecb4bd7c32dbe76a85554fdb4ba39592f87c92c541315a64fcbf27d422511"}
Oct 30 01:23:31.603526 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:31.603460    4393 scope.go:122] "RemoveContainer" containerID="ea7a05c5bd5662b0a96f950b902237681c33f95c0066beb4a4eec5e717690dd0"
Oct 30 01:23:31.614659 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:31.614637    4393 scope.go:122] "RemoveContainer" containerID="ea7a05c5bd5662b0a96f950b902237681c33f95c0066beb4a4eec5e717690dd0"
Oct 30 01:23:31.615005 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:23:31.614984    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea7a05c5bd5662b0a96f950b902237681c33f95c0066beb4a4eec5e717690dd0\": container with ID starting with ea7a05c5bd5662b0a96f950b902237681c33f95c0066beb4a4eec5e717690dd0 not found: ID does not exist" containerID="ea7a05c5bd5662b0a96f950b902237681c33f95c0066beb4a4eec5e717690dd0"
Oct 30 01:23:31.615093 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:31.615013    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea7a05c5bd5662b0a96f950b902237681c33f95c0066beb4a4eec5e717690dd0"} err="failed to get container status \"ea7a05c5bd5662b0a96f950b902237681c33f95c0066beb4a4eec5e717690dd0\": rpc error: code = NotFound desc = could not find container \"ea7a05c5bd5662b0a96f950b902237681c33f95c0066beb4a4eec5e717690dd0\": container with ID starting with ea7a05c5bd5662b0a96f950b902237681c33f95c0066beb4a4eec5e717690dd0 not found: ID does not exist"
Oct 30 01:23:31.653314 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:31.653276    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md"]
Oct 30 01:23:31.664700 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:31.664658    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-009e78-78cf5667f8-zt2md"]
Oct 30 01:23:31.751615 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:31.751516    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7028bbc0-dd7e-4525-a1db-8bf264d8ad58" path="/var/lib/kubelet/pods/7028bbc0-dd7e-4525-a1db-8bf264d8ad58/volumes"
Oct 30 01:23:32.299161 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:32.299110    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"]
Oct 30 01:23:32.299732 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:32.299343    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5" podUID="c026d820-c4ea-4d97-97b7-18757b273b8e" containerName="istio-proxy" containerID="cri-o://77d510a73c92dcdbf6c0a234a573c3a5e21dc5b1ad0925f347dfe43e5c70c051" gracePeriod=30
Oct 30 01:23:37.564532 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.564501    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"
Oct 30 01:23:37.596535 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.596450    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c026d820-c4ea-4d97-97b7-18757b273b8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c026d820-c4ea-4d97-97b7-18757b273b8e-workload-certs\") pod \"c026d820-c4ea-4d97-97b7-18757b273b8e\" (UID: \"c026d820-c4ea-4d97-97b7-18757b273b8e\") "
Oct 30 01:23:37.596535 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.596488    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c026d820-c4ea-4d97-97b7-18757b273b8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c026d820-c4ea-4d97-97b7-18757b273b8e-istio-data\") pod \"c026d820-c4ea-4d97-97b7-18757b273b8e\" (UID: \"c026d820-c4ea-4d97-97b7-18757b273b8e\") "
Oct 30 01:23:37.596821 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.596575    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c026d820-c4ea-4d97-97b7-18757b273b8e-kube-api-access-gntrr\" (UniqueName: \"kubernetes.io/projected/c026d820-c4ea-4d97-97b7-18757b273b8e-kube-api-access-gntrr\") pod \"c026d820-c4ea-4d97-97b7-18757b273b8e\" (UID: \"c026d820-c4ea-4d97-97b7-18757b273b8e\") "
Oct 30 01:23:37.596821 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.596619    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c026d820-c4ea-4d97-97b7-18757b273b8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c026d820-c4ea-4d97-97b7-18757b273b8e-istiod-ca-cert\") pod \"c026d820-c4ea-4d97-97b7-18757b273b8e\" (UID: \"c026d820-c4ea-4d97-97b7-18757b273b8e\") "
Oct 30 01:23:37.596821 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.596670    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c026d820-c4ea-4d97-97b7-18757b273b8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c026d820-c4ea-4d97-97b7-18757b273b8e-istio-podinfo\") pod \"c026d820-c4ea-4d97-97b7-18757b273b8e\" (UID: \"c026d820-c4ea-4d97-97b7-18757b273b8e\") "
Oct 30 01:23:37.596821 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.596694    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c026d820-c4ea-4d97-97b7-18757b273b8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c026d820-c4ea-4d97-97b7-18757b273b8e-istio-envoy\") pod \"c026d820-c4ea-4d97-97b7-18757b273b8e\" (UID: \"c026d820-c4ea-4d97-97b7-18757b273b8e\") "
Oct 30 01:23:37.596821 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.596730    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c026d820-c4ea-4d97-97b7-18757b273b8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c026d820-c4ea-4d97-97b7-18757b273b8e-credential-socket\") pod \"c026d820-c4ea-4d97-97b7-18757b273b8e\" (UID: \"c026d820-c4ea-4d97-97b7-18757b273b8e\") "
Oct 30 01:23:37.596821 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.596764    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c026d820-c4ea-4d97-97b7-18757b273b8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c026d820-c4ea-4d97-97b7-18757b273b8e-workload-socket\") pod \"c026d820-c4ea-4d97-97b7-18757b273b8e\" (UID: \"c026d820-c4ea-4d97-97b7-18757b273b8e\") "
Oct 30 01:23:37.597157 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.596833    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c026d820-c4ea-4d97-97b7-18757b273b8e-istio-token\" (UniqueName: \"kubernetes.io/projected/c026d820-c4ea-4d97-97b7-18757b273b8e-istio-token\") pod \"c026d820-c4ea-4d97-97b7-18757b273b8e\" (UID: \"c026d820-c4ea-4d97-97b7-18757b273b8e\") "
Oct 30 01:23:37.597157 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.596862    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c026d820-c4ea-4d97-97b7-18757b273b8e-istio-data" pod "c026d820-c4ea-4d97-97b7-18757b273b8e" (UID: "c026d820-c4ea-4d97-97b7-18757b273b8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:37.597157 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.597095    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c026d820-c4ea-4d97-97b7-18757b273b8e-credential-socket" pod "c026d820-c4ea-4d97-97b7-18757b273b8e" (UID: "c026d820-c4ea-4d97-97b7-18757b273b8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:37.597157 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.597122    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c026d820-c4ea-4d97-97b7-18757b273b8e-workload-socket" pod "c026d820-c4ea-4d97-97b7-18757b273b8e" (UID: "c026d820-c4ea-4d97-97b7-18757b273b8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:37.597368 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.597163    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c026d820-c4ea-4d97-97b7-18757b273b8e-istiod-ca-cert" pod "c026d820-c4ea-4d97-97b7-18757b273b8e" (UID: "c026d820-c4ea-4d97-97b7-18757b273b8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:37.597368 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.596765    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c026d820-c4ea-4d97-97b7-18757b273b8e-workload-certs" pod "c026d820-c4ea-4d97-97b7-18757b273b8e" (UID: "c026d820-c4ea-4d97-97b7-18757b273b8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:37.597368 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.597339    4393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c026d820-c4ea-4d97-97b7-18757b273b8e-workload-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:37.597368 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.597357    4393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c026d820-c4ea-4d97-97b7-18757b273b8e-istio-data\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:37.597368 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.597366    4393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c026d820-c4ea-4d97-97b7-18757b273b8e-istiod-ca-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:37.597825 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.597374    4393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c026d820-c4ea-4d97-97b7-18757b273b8e-credential-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:37.597825 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.597382    4393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c026d820-c4ea-4d97-97b7-18757b273b8e-workload-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:37.599199 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.599173    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c026d820-c4ea-4d97-97b7-18757b273b8e-istio-envoy" pod "c026d820-c4ea-4d97-97b7-18757b273b8e" (UID: "c026d820-c4ea-4d97-97b7-18757b273b8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:23:37.599345 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.599303    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c026d820-c4ea-4d97-97b7-18757b273b8e-istio-token" pod "c026d820-c4ea-4d97-97b7-18757b273b8e" (UID: "c026d820-c4ea-4d97-97b7-18757b273b8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:37.599606 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.599582    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c026d820-c4ea-4d97-97b7-18757b273b8e-istio-podinfo" pod "c026d820-c4ea-4d97-97b7-18757b273b8e" (UID: "c026d820-c4ea-4d97-97b7-18757b273b8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:23:37.599704 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.599620    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c026d820-c4ea-4d97-97b7-18757b273b8e-kube-api-access-gntrr" pod "c026d820-c4ea-4d97-97b7-18757b273b8e" (UID: "c026d820-c4ea-4d97-97b7-18757b273b8e"). InnerVolumeSpecName "kube-api-access-gntrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:37.645950 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.645910    4393 generic.go:356] "Generic (PLEG): container finished" podID="c026d820-c4ea-4d97-97b7-18757b273b8e" containerID="77d510a73c92dcdbf6c0a234a573c3a5e21dc5b1ad0925f347dfe43e5c70c051" exitCode=0
Oct 30 01:23:37.646153 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.645965    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"
Oct 30 01:23:37.646153 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.645994    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5" event={"ID":"c026d820-c4ea-4d97-97b7-18757b273b8e","Type":"ContainerDied","Data":"77d510a73c92dcdbf6c0a234a573c3a5e21dc5b1ad0925f347dfe43e5c70c051"}
Oct 30 01:23:37.646153 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.646074    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5" event={"ID":"c026d820-c4ea-4d97-97b7-18757b273b8e","Type":"ContainerDied","Data":"74662283cec3a65f3b1e92a89c013b6dfb0d45f53315cde49e2eca229d081c90"}
Oct 30 01:23:37.646153 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.646111    4393 scope.go:122] "RemoveContainer" containerID="77d510a73c92dcdbf6c0a234a573c3a5e21dc5b1ad0925f347dfe43e5c70c051"
Oct 30 01:23:37.661131 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.661107    4393 scope.go:122] "RemoveContainer" containerID="77d510a73c92dcdbf6c0a234a573c3a5e21dc5b1ad0925f347dfe43e5c70c051"
Oct 30 01:23:37.661481 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:23:37.661457    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77d510a73c92dcdbf6c0a234a573c3a5e21dc5b1ad0925f347dfe43e5c70c051\": container with ID starting with 77d510a73c92dcdbf6c0a234a573c3a5e21dc5b1ad0925f347dfe43e5c70c051 not found: ID does not exist" containerID="77d510a73c92dcdbf6c0a234a573c3a5e21dc5b1ad0925f347dfe43e5c70c051"
Oct 30 01:23:37.661603 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.661487    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77d510a73c92dcdbf6c0a234a573c3a5e21dc5b1ad0925f347dfe43e5c70c051"} err="failed to get container status \"77d510a73c92dcdbf6c0a234a573c3a5e21dc5b1ad0925f347dfe43e5c70c051\": rpc error: code = NotFound desc = could not find container \"77d510a73c92dcdbf6c0a234a573c3a5e21dc5b1ad0925f347dfe43e5c70c051\": container with ID starting with 77d510a73c92dcdbf6c0a234a573c3a5e21dc5b1ad0925f347dfe43e5c70c051 not found: ID does not exist"
Oct 30 01:23:37.696741 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.696703    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"]
Oct 30 01:23:37.698311 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.698288    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gntrr\" (UniqueName: \"kubernetes.io/projected/c026d820-c4ea-4d97-97b7-18757b273b8e-kube-api-access-gntrr\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:37.698377 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.698314    4393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c026d820-c4ea-4d97-97b7-18757b273b8e-istio-podinfo\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:37.698377 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.698331    4393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c026d820-c4ea-4d97-97b7-18757b273b8e-istio-envoy\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:37.698377 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.698342    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c026d820-c4ea-4d97-97b7-18757b273b8e-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:37.701141 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.701116    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca5363-openshift-default-6cfd85d975-fjbg5"]
Oct 30 01:23:37.724285 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:37.724250    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c026d820-c4ea-4d97-97b7-18757b273b8e" path="/var/lib/kubelet/pods/c026d820-c4ea-4d97-97b7-18757b273b8e/volumes"
Oct 30 01:23:42.206607 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:42.206559    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567"]
Oct 30 01:23:42.207141 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:42.206805    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567" podUID="473e63de-a201-41a6-b927-eee7643de4e1" containerName="maas-api" containerID="cri-o://4471345e9a58379ec1e35682083ebff8af939809aae778665a5b7ff812fa8ce1" gracePeriod=30
Oct 30 01:23:42.490591 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:42.490554    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567"
Oct 30 01:23:42.559357 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:42.559321    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/473e63de-a201-41a6-b927-eee7643de4e1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/473e63de-a201-41a6-b927-eee7643de4e1-openshift-service-ca\") pod \"473e63de-a201-41a6-b927-eee7643de4e1\" (UID: \"473e63de-a201-41a6-b927-eee7643de4e1\") "
Oct 30 01:23:42.559543 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:42.559372    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/473e63de-a201-41a6-b927-eee7643de4e1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/473e63de-a201-41a6-b927-eee7643de4e1-metrics-certs\") pod \"473e63de-a201-41a6-b927-eee7643de4e1\" (UID: \"473e63de-a201-41a6-b927-eee7643de4e1\") "
Oct 30 01:23:42.559543 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:42.559457    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/473e63de-a201-41a6-b927-eee7643de4e1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/473e63de-a201-41a6-b927-eee7643de4e1-maas-api-tls\") pod \"473e63de-a201-41a6-b927-eee7643de4e1\" (UID: \"473e63de-a201-41a6-b927-eee7643de4e1\") "
Oct 30 01:23:42.559688 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:42.559603    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/473e63de-a201-41a6-b927-eee7643de4e1-kube-api-access-f6b67\" (UniqueName: \"kubernetes.io/projected/473e63de-a201-41a6-b927-eee7643de4e1-kube-api-access-f6b67\") pod \"473e63de-a201-41a6-b927-eee7643de4e1\" (UID: \"473e63de-a201-41a6-b927-eee7643de4e1\") "
Oct 30 01:23:42.559821 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:42.559767    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/473e63de-a201-41a6-b927-eee7643de4e1-openshift-service-ca" pod "473e63de-a201-41a6-b927-eee7643de4e1" (UID: "473e63de-a201-41a6-b927-eee7643de4e1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:42.560133 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:42.560061    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/473e63de-a201-41a6-b927-eee7643de4e1-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:42.561923 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:42.561891    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/473e63de-a201-41a6-b927-eee7643de4e1-metrics-certs" pod "473e63de-a201-41a6-b927-eee7643de4e1" (UID: "473e63de-a201-41a6-b927-eee7643de4e1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:42.562594 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:42.562554    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/473e63de-a201-41a6-b927-eee7643de4e1-maas-api-tls" pod "473e63de-a201-41a6-b927-eee7643de4e1" (UID: "473e63de-a201-41a6-b927-eee7643de4e1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:42.562718 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:42.562660    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/473e63de-a201-41a6-b927-eee7643de4e1-kube-api-access-f6b67" pod "473e63de-a201-41a6-b927-eee7643de4e1" (UID: "473e63de-a201-41a6-b927-eee7643de4e1"). InnerVolumeSpecName "kube-api-access-f6b67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:42.661967 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:42.661902    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/473e63de-a201-41a6-b927-eee7643de4e1-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:42.661967 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:42.661970    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6b67\" (UniqueName: \"kubernetes.io/projected/473e63de-a201-41a6-b927-eee7643de4e1-kube-api-access-f6b67\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:42.662262 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:42.661988    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/473e63de-a201-41a6-b927-eee7643de4e1-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:42.682451 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:42.682411    4393 generic.go:356] "Generic (PLEG): container finished" podID="473e63de-a201-41a6-b927-eee7643de4e1" containerID="4471345e9a58379ec1e35682083ebff8af939809aae778665a5b7ff812fa8ce1" exitCode=0
Oct 30 01:23:42.682623 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:42.682468    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567"
Oct 30 01:23:42.682623 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:42.682487    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567" event={"ID":"473e63de-a201-41a6-b927-eee7643de4e1","Type":"ContainerDied","Data":"4471345e9a58379ec1e35682083ebff8af939809aae778665a5b7ff812fa8ce1"}
Oct 30 01:23:42.682623 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:42.682550    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567" event={"ID":"473e63de-a201-41a6-b927-eee7643de4e1","Type":"ContainerDied","Data":"ef86fe8c28126d5ee7e3744a27f4058af2971ab0da7c0de46be925bc3cab0891"}
Oct 30 01:23:42.682623 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:42.682587    4393 scope.go:122] "RemoveContainer" containerID="4471345e9a58379ec1e35682083ebff8af939809aae778665a5b7ff812fa8ce1"
Oct 30 01:23:42.693612 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:42.693589    4393 scope.go:122] "RemoveContainer" containerID="4471345e9a58379ec1e35682083ebff8af939809aae778665a5b7ff812fa8ce1"
Oct 30 01:23:42.693941 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:23:42.693922    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4471345e9a58379ec1e35682083ebff8af939809aae778665a5b7ff812fa8ce1\": container with ID starting with 4471345e9a58379ec1e35682083ebff8af939809aae778665a5b7ff812fa8ce1 not found: ID does not exist" containerID="4471345e9a58379ec1e35682083ebff8af939809aae778665a5b7ff812fa8ce1"
Oct 30 01:23:42.693989 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:42.693952    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4471345e9a58379ec1e35682083ebff8af939809aae778665a5b7ff812fa8ce1"} err="failed to get container status \"4471345e9a58379ec1e35682083ebff8af939809aae778665a5b7ff812fa8ce1\": rpc error: code = NotFound desc = could not find container \"4471345e9a58379ec1e35682083ebff8af939809aae778665a5b7ff812fa8ce1\": container with ID starting with 4471345e9a58379ec1e35682083ebff8af939809aae778665a5b7ff812fa8ce1 not found: ID does not exist"
Oct 30 01:23:42.734840 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:42.734734    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567"]
Oct 30 01:23:42.737859 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:42.737833    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a3619c-6cf6556b95-jb567"]
Oct 30 01:23:43.732586 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:43.732088    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="473e63de-a201-41a6-b927-eee7643de4e1" path="/var/lib/kubelet/pods/473e63de-a201-41a6-b927-eee7643de4e1/volumes"
Oct 30 01:23:49.700247 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:49.700204    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-7f92f6-5744474458-tvxhl"]
Oct 30 01:23:49.715066 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:49.715035    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm"]
Oct 30 01:23:49.715554 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:49.715519    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm" podUID="6da14397-0216-4ada-83bf-72addba282e5" containerName="maas-api" containerID="cri-o://372a4304cb68e907993b4246e0387ce0ef6354e4e02fce81581686e46e325d45" gracePeriod=30
Oct 30 01:23:49.751404 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:49.748839    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7f92f6-79df4f4c6c-c4qwp"]
Oct 30 01:23:49.978856 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:49.978829    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm"
Oct 30 01:23:50.052149 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:50.052117    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6da14397-0216-4ada-83bf-72addba282e5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6da14397-0216-4ada-83bf-72addba282e5-maas-api-tls\") pod \"6da14397-0216-4ada-83bf-72addba282e5\" (UID: \"6da14397-0216-4ada-83bf-72addba282e5\") "
Oct 30 01:23:50.052338 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:50.052231    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6da14397-0216-4ada-83bf-72addba282e5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6da14397-0216-4ada-83bf-72addba282e5-openshift-service-ca\") pod \"6da14397-0216-4ada-83bf-72addba282e5\" (UID: \"6da14397-0216-4ada-83bf-72addba282e5\") "
Oct 30 01:23:50.052338 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:50.052268    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6da14397-0216-4ada-83bf-72addba282e5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6da14397-0216-4ada-83bf-72addba282e5-metrics-certs\") pod \"6da14397-0216-4ada-83bf-72addba282e5\" (UID: \"6da14397-0216-4ada-83bf-72addba282e5\") "
Oct 30 01:23:50.052477 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:50.052407    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6da14397-0216-4ada-83bf-72addba282e5-kube-api-access-px8k4\" (UniqueName: \"kubernetes.io/projected/6da14397-0216-4ada-83bf-72addba282e5-kube-api-access-px8k4\") pod \"6da14397-0216-4ada-83bf-72addba282e5\" (UID: \"6da14397-0216-4ada-83bf-72addba282e5\") "
Oct 30 01:23:50.052715 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:50.052651    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6da14397-0216-4ada-83bf-72addba282e5-openshift-service-ca" pod "6da14397-0216-4ada-83bf-72addba282e5" (UID: "6da14397-0216-4ada-83bf-72addba282e5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:50.052895 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:50.052876    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6da14397-0216-4ada-83bf-72addba282e5-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:50.054572 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:50.054529    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6da14397-0216-4ada-83bf-72addba282e5-metrics-certs" pod "6da14397-0216-4ada-83bf-72addba282e5" (UID: "6da14397-0216-4ada-83bf-72addba282e5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:50.054665 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:50.054622    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6da14397-0216-4ada-83bf-72addba282e5-maas-api-tls" pod "6da14397-0216-4ada-83bf-72addba282e5" (UID: "6da14397-0216-4ada-83bf-72addba282e5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:23:50.055020 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:50.054996    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6da14397-0216-4ada-83bf-72addba282e5-kube-api-access-px8k4" pod "6da14397-0216-4ada-83bf-72addba282e5" (UID: "6da14397-0216-4ada-83bf-72addba282e5"). InnerVolumeSpecName "kube-api-access-px8k4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:50.153848 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:50.153714    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6da14397-0216-4ada-83bf-72addba282e5-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:50.153848 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:50.153760    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px8k4\" (UniqueName: \"kubernetes.io/projected/6da14397-0216-4ada-83bf-72addba282e5-kube-api-access-px8k4\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:50.153848 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:50.153793    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6da14397-0216-4ada-83bf-72addba282e5-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:50.758265 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:50.758232    4393 generic.go:356] "Generic (PLEG): container finished" podID="6da14397-0216-4ada-83bf-72addba282e5" containerID="372a4304cb68e907993b4246e0387ce0ef6354e4e02fce81581686e46e325d45" exitCode=0
Oct 30 01:23:50.758765 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:50.758290    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm"
Oct 30 01:23:50.758765 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:50.758295    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm" event={"ID":"6da14397-0216-4ada-83bf-72addba282e5","Type":"ContainerDied","Data":"372a4304cb68e907993b4246e0387ce0ef6354e4e02fce81581686e46e325d45"}
Oct 30 01:23:50.758765 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:50.758355    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm" event={"ID":"6da14397-0216-4ada-83bf-72addba282e5","Type":"ContainerDied","Data":"db4d4f671e46ab0a063eafacd79847b42a0e8535681b9d8ee95eb38a01422fde"}
Oct 30 01:23:50.758765 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:50.758380    4393 scope.go:122] "RemoveContainer" containerID="372a4304cb68e907993b4246e0387ce0ef6354e4e02fce81581686e46e325d45"
Oct 30 01:23:50.769862 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:50.769838    4393 scope.go:122] "RemoveContainer" containerID="372a4304cb68e907993b4246e0387ce0ef6354e4e02fce81581686e46e325d45"
Oct 30 01:23:50.770179 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:23:50.770160    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"372a4304cb68e907993b4246e0387ce0ef6354e4e02fce81581686e46e325d45\": container with ID starting with 372a4304cb68e907993b4246e0387ce0ef6354e4e02fce81581686e46e325d45 not found: ID does not exist" containerID="372a4304cb68e907993b4246e0387ce0ef6354e4e02fce81581686e46e325d45"
Oct 30 01:23:50.770247 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:50.770187    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"372a4304cb68e907993b4246e0387ce0ef6354e4e02fce81581686e46e325d45"} err="failed to get container status \"372a4304cb68e907993b4246e0387ce0ef6354e4e02fce81581686e46e325d45\": rpc error: code = NotFound desc = could not find container \"372a4304cb68e907993b4246e0387ce0ef6354e4e02fce81581686e46e325d45\": container with ID starting with 372a4304cb68e907993b4246e0387ce0ef6354e4e02fce81581686e46e325d45 not found: ID does not exist"
Oct 30 01:23:50.807758 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:50.807719    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm"]
Oct 30 01:23:50.811404 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:50.811376    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-7f92f6-5b9469d9f6-792mm"]
Oct 30 01:23:51.723352 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:51.723308    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6da14397-0216-4ada-83bf-72addba282e5" path="/var/lib/kubelet/pods/6da14397-0216-4ada-83bf-72addba282e5/volumes"
Oct 30 01:23:54.700804 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:54.700712    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-7f92f6-5744474458-tvxhl" podUID="a674fcd8-3f60-4f70-b199-a2ad09d74fb2" containerName="payload-processing" containerID="cri-o://fffac304f5df1d0f08fbbeb0ba526f0cfd4152caaee1dab2ec0906de2188b634" gracePeriod=25
Oct 30 01:23:54.749919 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:54.749877    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7f92f6-79df4f4c6c-c4qwp" podUID="4debacaf-9ed1-4e85-a8f3-4f52a76ef705" containerName="payload-pre-processing" containerID="cri-o://e81b1711ae3e8fccc1eab05ec101393d81a36a37e0873c67f5b288902a41e1b4" gracePeriod=25
Oct 30 01:23:55.037971 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:55.037941    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-7f92f6-5744474458-tvxhl"
Oct 30 01:23:55.042108 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:55.042075    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7f92f6-79df4f4c6c-c4qwp"
Oct 30 01:23:55.204744 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:55.204702    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4debacaf-9ed1-4e85-a8f3-4f52a76ef705-kube-api-access-ctm9l\" (UniqueName: \"kubernetes.io/projected/4debacaf-9ed1-4e85-a8f3-4f52a76ef705-kube-api-access-ctm9l\") pod \"4debacaf-9ed1-4e85-a8f3-4f52a76ef705\" (UID: \"4debacaf-9ed1-4e85-a8f3-4f52a76ef705\") "
Oct 30 01:23:55.204955 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:55.204843    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a674fcd8-3f60-4f70-b199-a2ad09d74fb2-kube-api-access-5slr7\" (UniqueName: \"kubernetes.io/projected/a674fcd8-3f60-4f70-b199-a2ad09d74fb2-kube-api-access-5slr7\") pod \"a674fcd8-3f60-4f70-b199-a2ad09d74fb2\" (UID: \"a674fcd8-3f60-4f70-b199-a2ad09d74fb2\") "
Oct 30 01:23:55.204955 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:55.204902    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4debacaf-9ed1-4e85-a8f3-4f52a76ef705-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4debacaf-9ed1-4e85-a8f3-4f52a76ef705-plugins-config-volume\") pod \"4debacaf-9ed1-4e85-a8f3-4f52a76ef705\" (UID: \"4debacaf-9ed1-4e85-a8f3-4f52a76ef705\") "
Oct 30 01:23:55.205105 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:55.204978    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a674fcd8-3f60-4f70-b199-a2ad09d74fb2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a674fcd8-3f60-4f70-b199-a2ad09d74fb2-plugins-config-volume\") pod \"a674fcd8-3f60-4f70-b199-a2ad09d74fb2\" (UID: \"a674fcd8-3f60-4f70-b199-a2ad09d74fb2\") "
Oct 30 01:23:55.205352 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:55.205325    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4debacaf-9ed1-4e85-a8f3-4f52a76ef705-plugins-config-volume" pod "4debacaf-9ed1-4e85-a8f3-4f52a76ef705" (UID: "4debacaf-9ed1-4e85-a8f3-4f52a76ef705"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:55.205431 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:55.205386    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a674fcd8-3f60-4f70-b199-a2ad09d74fb2-plugins-config-volume" pod "a674fcd8-3f60-4f70-b199-a2ad09d74fb2" (UID: "a674fcd8-3f60-4f70-b199-a2ad09d74fb2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:23:55.207314 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:55.207291    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4debacaf-9ed1-4e85-a8f3-4f52a76ef705-kube-api-access-ctm9l" pod "4debacaf-9ed1-4e85-a8f3-4f52a76ef705" (UID: "4debacaf-9ed1-4e85-a8f3-4f52a76ef705"). InnerVolumeSpecName "kube-api-access-ctm9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:55.207367 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:55.207291    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a674fcd8-3f60-4f70-b199-a2ad09d74fb2-kube-api-access-5slr7" pod "a674fcd8-3f60-4f70-b199-a2ad09d74fb2" (UID: "a674fcd8-3f60-4f70-b199-a2ad09d74fb2"). InnerVolumeSpecName "kube-api-access-5slr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:23:55.307048 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:55.307004    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a674fcd8-3f60-4f70-b199-a2ad09d74fb2-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:55.307048 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:55.307042    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctm9l\" (UniqueName: \"kubernetes.io/projected/4debacaf-9ed1-4e85-a8f3-4f52a76ef705-kube-api-access-ctm9l\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:55.307048 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:55.307051    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5slr7\" (UniqueName: \"kubernetes.io/projected/a674fcd8-3f60-4f70-b199-a2ad09d74fb2-kube-api-access-5slr7\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:55.307311 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:55.307060    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4debacaf-9ed1-4e85-a8f3-4f52a76ef705-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:23:55.792575 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:55.792535    4393 generic.go:356] "Generic (PLEG): container finished" podID="4debacaf-9ed1-4e85-a8f3-4f52a76ef705" containerID="e81b1711ae3e8fccc1eab05ec101393d81a36a37e0873c67f5b288902a41e1b4" exitCode=0
Oct 30 01:23:55.793179 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:55.792587    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7f92f6-79df4f4c6c-c4qwp"
Oct 30 01:23:55.793179 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:55.792623    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7f92f6-79df4f4c6c-c4qwp" event={"ID":"4debacaf-9ed1-4e85-a8f3-4f52a76ef705","Type":"ContainerDied","Data":"e81b1711ae3e8fccc1eab05ec101393d81a36a37e0873c67f5b288902a41e1b4"}
Oct 30 01:23:55.793179 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:55.792692    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7f92f6-79df4f4c6c-c4qwp" event={"ID":"4debacaf-9ed1-4e85-a8f3-4f52a76ef705","Type":"ContainerDied","Data":"8fcacb0afbcf8b29dae3505e62d84b1771ce87a73a5c7f6dc4a8afb02aee824d"}
Oct 30 01:23:55.793179 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:55.792726    4393 scope.go:122] "RemoveContainer" containerID="e81b1711ae3e8fccc1eab05ec101393d81a36a37e0873c67f5b288902a41e1b4"
Oct 30 01:23:55.794722 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:55.794694    4393 generic.go:356] "Generic (PLEG): container finished" podID="a674fcd8-3f60-4f70-b199-a2ad09d74fb2" containerID="fffac304f5df1d0f08fbbeb0ba526f0cfd4152caaee1dab2ec0906de2188b634" exitCode=0
Oct 30 01:23:55.794885 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:55.794839    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-7f92f6-5744474458-tvxhl" event={"ID":"a674fcd8-3f60-4f70-b199-a2ad09d74fb2","Type":"ContainerDied","Data":"fffac304f5df1d0f08fbbeb0ba526f0cfd4152caaee1dab2ec0906de2188b634"}
Oct 30 01:23:55.794961 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:55.794886    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-7f92f6-5744474458-tvxhl" event={"ID":"a674fcd8-3f60-4f70-b199-a2ad09d74fb2","Type":"ContainerDied","Data":"bb751936dcbfbe905c42186e264b539ba33387acbd8e8cd335850e1b521ea61e"}
Oct 30 01:23:55.794961 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:55.794879    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-7f92f6-5744474458-tvxhl"
Oct 30 01:23:55.804287 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:55.804262    4393 scope.go:122] "RemoveContainer" containerID="e81b1711ae3e8fccc1eab05ec101393d81a36a37e0873c67f5b288902a41e1b4"
Oct 30 01:23:55.804603 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:23:55.804577    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e81b1711ae3e8fccc1eab05ec101393d81a36a37e0873c67f5b288902a41e1b4\": container with ID starting with e81b1711ae3e8fccc1eab05ec101393d81a36a37e0873c67f5b288902a41e1b4 not found: ID does not exist" containerID="e81b1711ae3e8fccc1eab05ec101393d81a36a37e0873c67f5b288902a41e1b4"
Oct 30 01:23:55.804692 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:55.804608    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e81b1711ae3e8fccc1eab05ec101393d81a36a37e0873c67f5b288902a41e1b4"} err="failed to get container status \"e81b1711ae3e8fccc1eab05ec101393d81a36a37e0873c67f5b288902a41e1b4\": rpc error: code = NotFound desc = could not find container \"e81b1711ae3e8fccc1eab05ec101393d81a36a37e0873c67f5b288902a41e1b4\": container with ID starting with e81b1711ae3e8fccc1eab05ec101393d81a36a37e0873c67f5b288902a41e1b4 not found: ID does not exist"
Oct 30 01:23:55.804692 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:55.804627    4393 scope.go:122] "RemoveContainer" containerID="fffac304f5df1d0f08fbbeb0ba526f0cfd4152caaee1dab2ec0906de2188b634"
Oct 30 01:23:55.815513 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:55.815494    4393 scope.go:122] "RemoveContainer" containerID="fffac304f5df1d0f08fbbeb0ba526f0cfd4152caaee1dab2ec0906de2188b634"
Oct 30 01:23:55.815862 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:23:55.815836    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fffac304f5df1d0f08fbbeb0ba526f0cfd4152caaee1dab2ec0906de2188b634\": container with ID starting with fffac304f5df1d0f08fbbeb0ba526f0cfd4152caaee1dab2ec0906de2188b634 not found: ID does not exist" containerID="fffac304f5df1d0f08fbbeb0ba526f0cfd4152caaee1dab2ec0906de2188b634"
Oct 30 01:23:55.815946 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:55.815868    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fffac304f5df1d0f08fbbeb0ba526f0cfd4152caaee1dab2ec0906de2188b634"} err="failed to get container status \"fffac304f5df1d0f08fbbeb0ba526f0cfd4152caaee1dab2ec0906de2188b634\": rpc error: code = NotFound desc = could not find container \"fffac304f5df1d0f08fbbeb0ba526f0cfd4152caaee1dab2ec0906de2188b634\": container with ID starting with fffac304f5df1d0f08fbbeb0ba526f0cfd4152caaee1dab2ec0906de2188b634 not found: ID does not exist"
Oct 30 01:23:55.893520 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:55.893481    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-7f92f6-5744474458-tvxhl"]
Oct 30 01:23:55.904991 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:55.904958    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-7f92f6-5744474458-tvxhl"]
Oct 30 01:23:55.952820 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:55.952749    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7f92f6-79df4f4c6c-c4qwp"]
Oct 30 01:23:55.956354 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:55.956323    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-7f92f6-79df4f4c6c-c4qwp"]
Oct 30 01:23:56.511160 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:23:56.511125    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:23:57.724584 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:57.724542    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4debacaf-9ed1-4e85-a8f3-4f52a76ef705" path="/var/lib/kubelet/pods/4debacaf-9ed1-4e85-a8f3-4f52a76ef705/volumes"
Oct 30 01:23:57.725113 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:57.725097    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a674fcd8-3f60-4f70-b199-a2ad09d74fb2" path="/var/lib/kubelet/pods/a674fcd8-3f60-4f70-b199-a2ad09d74fb2/volumes"
Oct 30 01:23:58.274169 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:58.274128    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"]
Oct 30 01:23:58.274455 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:23:58.274404    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4" podUID="07623c58-4919-43fe-ad87-6e47c6af9596" containerName="istio-proxy" containerID="cri-o://11c43132f9f9288dd204660a7f2b236d8dfd20585d9301d92c58b2533f39a6b2" gracePeriod=30
Oct 30 01:24:01.729854 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:01.729820    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-1a11cf-b489f44f9-lwz7b"]
Oct 30 01:24:03.547679 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.547602    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"
Oct 30 01:24:03.604517 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.604469    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07623c58-4919-43fe-ad87-6e47c6af9596-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07623c58-4919-43fe-ad87-6e47c6af9596-workload-certs\") pod \"07623c58-4919-43fe-ad87-6e47c6af9596\" (UID: \"07623c58-4919-43fe-ad87-6e47c6af9596\") "
Oct 30 01:24:03.604732 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.604552    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07623c58-4919-43fe-ad87-6e47c6af9596-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07623c58-4919-43fe-ad87-6e47c6af9596-istio-podinfo\") pod \"07623c58-4919-43fe-ad87-6e47c6af9596\" (UID: \"07623c58-4919-43fe-ad87-6e47c6af9596\") "
Oct 30 01:24:03.604732 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.604608    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07623c58-4919-43fe-ad87-6e47c6af9596-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07623c58-4919-43fe-ad87-6e47c6af9596-istio-envoy\") pod \"07623c58-4919-43fe-ad87-6e47c6af9596\" (UID: \"07623c58-4919-43fe-ad87-6e47c6af9596\") "
Oct 30 01:24:03.604732 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.604691    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07623c58-4919-43fe-ad87-6e47c6af9596-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07623c58-4919-43fe-ad87-6e47c6af9596-credential-socket\") pod \"07623c58-4919-43fe-ad87-6e47c6af9596\" (UID: \"07623c58-4919-43fe-ad87-6e47c6af9596\") "
Oct 30 01:24:03.604732 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.604718    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07623c58-4919-43fe-ad87-6e47c6af9596-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07623c58-4919-43fe-ad87-6e47c6af9596-workload-socket\") pod \"07623c58-4919-43fe-ad87-6e47c6af9596\" (UID: \"07623c58-4919-43fe-ad87-6e47c6af9596\") "
Oct 30 01:24:03.605012 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.604755    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07623c58-4919-43fe-ad87-6e47c6af9596-kube-api-access-4czjt\" (UniqueName: \"kubernetes.io/projected/07623c58-4919-43fe-ad87-6e47c6af9596-kube-api-access-4czjt\") pod \"07623c58-4919-43fe-ad87-6e47c6af9596\" (UID: \"07623c58-4919-43fe-ad87-6e47c6af9596\") "
Oct 30 01:24:03.605226 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.605166    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07623c58-4919-43fe-ad87-6e47c6af9596-workload-socket" pod "07623c58-4919-43fe-ad87-6e47c6af9596" (UID: "07623c58-4919-43fe-ad87-6e47c6af9596"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:03.605393 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.605339    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07623c58-4919-43fe-ad87-6e47c6af9596-credential-socket" pod "07623c58-4919-43fe-ad87-6e47c6af9596" (UID: "07623c58-4919-43fe-ad87-6e47c6af9596"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:03.605601 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.605566    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07623c58-4919-43fe-ad87-6e47c6af9596-workload-certs" pod "07623c58-4919-43fe-ad87-6e47c6af9596" (UID: "07623c58-4919-43fe-ad87-6e47c6af9596"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:03.608526 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.608483    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07623c58-4919-43fe-ad87-6e47c6af9596-istio-podinfo" pod "07623c58-4919-43fe-ad87-6e47c6af9596" (UID: "07623c58-4919-43fe-ad87-6e47c6af9596"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:24:03.608749 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.608728    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07623c58-4919-43fe-ad87-6e47c6af9596-istio-envoy" pod "07623c58-4919-43fe-ad87-6e47c6af9596" (UID: "07623c58-4919-43fe-ad87-6e47c6af9596"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:03.610849 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.610823    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07623c58-4919-43fe-ad87-6e47c6af9596-kube-api-access-4czjt" pod "07623c58-4919-43fe-ad87-6e47c6af9596" (UID: "07623c58-4919-43fe-ad87-6e47c6af9596"). InnerVolumeSpecName "kube-api-access-4czjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:03.706167 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.706128    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07623c58-4919-43fe-ad87-6e47c6af9596-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07623c58-4919-43fe-ad87-6e47c6af9596-istiod-ca-cert\") pod \"07623c58-4919-43fe-ad87-6e47c6af9596\" (UID: \"07623c58-4919-43fe-ad87-6e47c6af9596\") "
Oct 30 01:24:03.706351 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.706183    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07623c58-4919-43fe-ad87-6e47c6af9596-istio-token\" (UniqueName: \"kubernetes.io/projected/07623c58-4919-43fe-ad87-6e47c6af9596-istio-token\") pod \"07623c58-4919-43fe-ad87-6e47c6af9596\" (UID: \"07623c58-4919-43fe-ad87-6e47c6af9596\") "
Oct 30 01:24:03.706351 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.706218    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07623c58-4919-43fe-ad87-6e47c6af9596-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07623c58-4919-43fe-ad87-6e47c6af9596-istio-data\") pod \"07623c58-4919-43fe-ad87-6e47c6af9596\" (UID: \"07623c58-4919-43fe-ad87-6e47c6af9596\") "
Oct 30 01:24:03.706493 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.706480    4393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07623c58-4919-43fe-ad87-6e47c6af9596-credential-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:03.706535 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.706497    4393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07623c58-4919-43fe-ad87-6e47c6af9596-workload-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:03.706535 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.706506    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4czjt\" (UniqueName: \"kubernetes.io/projected/07623c58-4919-43fe-ad87-6e47c6af9596-kube-api-access-4czjt\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:03.706535 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.706516    4393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07623c58-4919-43fe-ad87-6e47c6af9596-workload-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:03.706535 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.706524    4393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07623c58-4919-43fe-ad87-6e47c6af9596-istio-podinfo\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:03.706535 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.706532    4393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07623c58-4919-43fe-ad87-6e47c6af9596-istio-envoy\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:03.706727 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.706523    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07623c58-4919-43fe-ad87-6e47c6af9596-istiod-ca-cert" pod "07623c58-4919-43fe-ad87-6e47c6af9596" (UID: "07623c58-4919-43fe-ad87-6e47c6af9596"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:03.706727 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.706610    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07623c58-4919-43fe-ad87-6e47c6af9596-istio-data" pod "07623c58-4919-43fe-ad87-6e47c6af9596" (UID: "07623c58-4919-43fe-ad87-6e47c6af9596"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:03.708580 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.708557    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07623c58-4919-43fe-ad87-6e47c6af9596-istio-token" pod "07623c58-4919-43fe-ad87-6e47c6af9596" (UID: "07623c58-4919-43fe-ad87-6e47c6af9596"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:03.724409 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.724380    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"]
Oct 30 01:24:03.807943 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.807847    4393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07623c58-4919-43fe-ad87-6e47c6af9596-istiod-ca-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:03.807943 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.807878    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07623c58-4919-43fe-ad87-6e47c6af9596-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:03.807943 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.807887    4393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07623c58-4919-43fe-ad87-6e47c6af9596-istio-data\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:03.848319 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.848264    4393 generic.go:356] "Generic (PLEG): container finished" podID="07623c58-4919-43fe-ad87-6e47c6af9596" containerID="11c43132f9f9288dd204660a7f2b236d8dfd20585d9301d92c58b2533f39a6b2" exitCode=0
Oct 30 01:24:03.848533 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.848351    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"
Oct 30 01:24:03.848533 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.848400    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4" event={"ID":"07623c58-4919-43fe-ad87-6e47c6af9596","Type":"ContainerDied","Data":"11c43132f9f9288dd204660a7f2b236d8dfd20585d9301d92c58b2533f39a6b2"}
Oct 30 01:24:03.848533 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.848444    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4" event={"ID":"07623c58-4919-43fe-ad87-6e47c6af9596","Type":"ContainerDied","Data":"21edc06989304625a97153a41baa07e3e5433d097e982abfb56da102fc8e29a2"}
Oct 30 01:24:03.848533 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.848473    4393 scope.go:122] "RemoveContainer" containerID="11c43132f9f9288dd204660a7f2b236d8dfd20585d9301d92c58b2533f39a6b2"
Oct 30 01:24:03.859388 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.859359    4393 scope.go:122] "RemoveContainer" containerID="11c43132f9f9288dd204660a7f2b236d8dfd20585d9301d92c58b2533f39a6b2"
Oct 30 01:24:03.859743 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:24:03.859719    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11c43132f9f9288dd204660a7f2b236d8dfd20585d9301d92c58b2533f39a6b2\": container with ID starting with 11c43132f9f9288dd204660a7f2b236d8dfd20585d9301d92c58b2533f39a6b2 not found: ID does not exist" containerID="11c43132f9f9288dd204660a7f2b236d8dfd20585d9301d92c58b2533f39a6b2"
Oct 30 01:24:03.859854 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.859752    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11c43132f9f9288dd204660a7f2b236d8dfd20585d9301d92c58b2533f39a6b2"} err="failed to get container status \"11c43132f9f9288dd204660a7f2b236d8dfd20585d9301d92c58b2533f39a6b2\": rpc error: code = NotFound desc = could not find container \"11c43132f9f9288dd204660a7f2b236d8dfd20585d9301d92c58b2533f39a6b2\": container with ID starting with 11c43132f9f9288dd204660a7f2b236d8dfd20585d9301d92c58b2533f39a6b2 not found: ID does not exist"
Oct 30 01:24:03.899987 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.899954    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"]
Oct 30 01:24:03.903164 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:03.903137    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-7f92f6-openshift-default-85bfc49969-m9zn4"]
Oct 30 01:24:05.723928 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:05.723889    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07623c58-4919-43fe-ad87-6e47c6af9596" path="/var/lib/kubelet/pods/07623c58-4919-43fe-ad87-6e47c6af9596/volumes"
Oct 30 01:24:06.730433 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:06.730381    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1a11cf-b489f44f9-lwz7b" podUID="1d8f254c-2610-4cb4-8e28-a59aea8a6de9" containerName="payload-pre-processing" containerID="cri-o://694a819e0ecd39bfb183417f157f995d61297b94a83e1dd353480a0c0a3a3246" gracePeriod=25
Oct 30 01:24:11.272332 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:11.272293    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1a11cf-b489f44f9-lwz7b" podUID="1d8f254c-2610-4cb4-8e28-a59aea8a6de9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:24:16.429939 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:16.429901    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw"]
Oct 30 01:24:16.430639 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:16.430603    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw" podUID="4ea0d2c2-401c-4873-ac44-75481cdfd848" containerName="maas-api" containerID="cri-o://4402469f007f85b7fb884e5835052e3291b26eff7ed194fc769b01b5f87d5d9f" gracePeriod=30
Oct 30 01:24:16.720746 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:16.720717    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw"
Oct 30 01:24:16.877260 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:16.877221    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ea0d2c2-401c-4873-ac44-75481cdfd848-kube-api-access-46d55\" (UniqueName: \"kubernetes.io/projected/4ea0d2c2-401c-4873-ac44-75481cdfd848-kube-api-access-46d55\") pod \"4ea0d2c2-401c-4873-ac44-75481cdfd848\" (UID: \"4ea0d2c2-401c-4873-ac44-75481cdfd848\") "
Oct 30 01:24:16.877464 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:16.877349    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ea0d2c2-401c-4873-ac44-75481cdfd848-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ea0d2c2-401c-4873-ac44-75481cdfd848-maas-api-tls\") pod \"4ea0d2c2-401c-4873-ac44-75481cdfd848\" (UID: \"4ea0d2c2-401c-4873-ac44-75481cdfd848\") "
Oct 30 01:24:16.877464 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:16.877402    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ea0d2c2-401c-4873-ac44-75481cdfd848-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea0d2c2-401c-4873-ac44-75481cdfd848-openshift-service-ca\") pod \"4ea0d2c2-401c-4873-ac44-75481cdfd848\" (UID: \"4ea0d2c2-401c-4873-ac44-75481cdfd848\") "
Oct 30 01:24:16.877464 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:16.877452    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ea0d2c2-401c-4873-ac44-75481cdfd848-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ea0d2c2-401c-4873-ac44-75481cdfd848-metrics-certs\") pod \"4ea0d2c2-401c-4873-ac44-75481cdfd848\" (UID: \"4ea0d2c2-401c-4873-ac44-75481cdfd848\") "
Oct 30 01:24:16.878021 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:16.877987    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ea0d2c2-401c-4873-ac44-75481cdfd848-openshift-service-ca" pod "4ea0d2c2-401c-4873-ac44-75481cdfd848" (UID: "4ea0d2c2-401c-4873-ac44-75481cdfd848"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:16.879758 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:16.879725    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ea0d2c2-401c-4873-ac44-75481cdfd848-maas-api-tls" pod "4ea0d2c2-401c-4873-ac44-75481cdfd848" (UID: "4ea0d2c2-401c-4873-ac44-75481cdfd848"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:16.879917 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:16.879759    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea0d2c2-401c-4873-ac44-75481cdfd848-kube-api-access-46d55" pod "4ea0d2c2-401c-4873-ac44-75481cdfd848" (UID: "4ea0d2c2-401c-4873-ac44-75481cdfd848"). InnerVolumeSpecName "kube-api-access-46d55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:16.880003 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:16.879975    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ea0d2c2-401c-4873-ac44-75481cdfd848-metrics-certs" pod "4ea0d2c2-401c-4873-ac44-75481cdfd848" (UID: "4ea0d2c2-401c-4873-ac44-75481cdfd848"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:16.928042 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:16.928006    4393 generic.go:356] "Generic (PLEG): container finished" podID="4ea0d2c2-401c-4873-ac44-75481cdfd848" containerID="4402469f007f85b7fb884e5835052e3291b26eff7ed194fc769b01b5f87d5d9f" exitCode=0
Oct 30 01:24:16.928231 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:16.928055    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw"
Oct 30 01:24:16.928231 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:16.928094    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw" event={"ID":"4ea0d2c2-401c-4873-ac44-75481cdfd848","Type":"ContainerDied","Data":"4402469f007f85b7fb884e5835052e3291b26eff7ed194fc769b01b5f87d5d9f"}
Oct 30 01:24:16.928231 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:16.928150    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw" event={"ID":"4ea0d2c2-401c-4873-ac44-75481cdfd848","Type":"ContainerDied","Data":"34906018c61a314c14225b5096f45ea883251df504e410f3f139304c1e5a69e9"}
Oct 30 01:24:16.928231 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:16.928177    4393 scope.go:122] "RemoveContainer" containerID="4402469f007f85b7fb884e5835052e3291b26eff7ed194fc769b01b5f87d5d9f"
Oct 30 01:24:16.942395 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:16.942370    4393 scope.go:122] "RemoveContainer" containerID="4402469f007f85b7fb884e5835052e3291b26eff7ed194fc769b01b5f87d5d9f"
Oct 30 01:24:16.942856 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:24:16.942827    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4402469f007f85b7fb884e5835052e3291b26eff7ed194fc769b01b5f87d5d9f\": container with ID starting with 4402469f007f85b7fb884e5835052e3291b26eff7ed194fc769b01b5f87d5d9f not found: ID does not exist" containerID="4402469f007f85b7fb884e5835052e3291b26eff7ed194fc769b01b5f87d5d9f"
Oct 30 01:24:16.942942 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:16.942865    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4402469f007f85b7fb884e5835052e3291b26eff7ed194fc769b01b5f87d5d9f"} err="failed to get container status \"4402469f007f85b7fb884e5835052e3291b26eff7ed194fc769b01b5f87d5d9f\": rpc error: code = NotFound desc = could not find container \"4402469f007f85b7fb884e5835052e3291b26eff7ed194fc769b01b5f87d5d9f\": container with ID starting with 4402469f007f85b7fb884e5835052e3291b26eff7ed194fc769b01b5f87d5d9f not found: ID does not exist"
Oct 30 01:24:16.978276 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:16.978215    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw"]
Oct 30 01:24:16.978750 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:16.978723    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ea0d2c2-401c-4873-ac44-75481cdfd848-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:16.978848 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:16.978754    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea0d2c2-401c-4873-ac44-75481cdfd848-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:16.978848 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:16.978767    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ea0d2c2-401c-4873-ac44-75481cdfd848-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:16.978848 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:16.978806    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46d55\" (UniqueName: \"kubernetes.io/projected/4ea0d2c2-401c-4873-ac44-75481cdfd848-kube-api-access-46d55\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:16.982464 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:16.982433    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-1a11cf-5549678f4c-mzfpw"]
Oct 30 01:24:17.725183 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:17.725146    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ea0d2c2-401c-4873-ac44-75481cdfd848" path="/var/lib/kubelet/pods/4ea0d2c2-401c-4873-ac44-75481cdfd848/volumes"
Oct 30 01:24:18.747771 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:18.747730    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x" podUID="28f3e4ff-5875-4e5d-8cab-60d1a5ba97be" containerName="main" containerID="cri-o://2f989378070201471da6c09ec75f086bafd5930ca1f2003b959fc4103c2a7c3b" gracePeriod=45
Oct 30 01:24:18.947413 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:18.947373    4393 generic.go:356] "Generic (PLEG): container finished" podID="28f3e4ff-5875-4e5d-8cab-60d1a5ba97be" containerID="2f989378070201471da6c09ec75f086bafd5930ca1f2003b959fc4103c2a7c3b" exitCode=0
Oct 30 01:24:18.947612 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:18.947590    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x" event={"ID":"28f3e4ff-5875-4e5d-8cab-60d1a5ba97be","Type":"ContainerDied","Data":"2f989378070201471da6c09ec75f086bafd5930ca1f2003b959fc4103c2a7c3b"}
Oct 30 01:24:19.009560 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:19.009489    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"
Oct 30 01:24:19.099625 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:19.099584    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-tmp-dir\") pod \"28f3e4ff-5875-4e5d-8cab-60d1a5ba97be\" (UID: \"28f3e4ff-5875-4e5d-8cab-60d1a5ba97be\") "
Oct 30 01:24:19.099932 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:19.099648    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-tls-certs\" (UniqueName: \"kubernetes.io/secret/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-tls-certs\") pod \"28f3e4ff-5875-4e5d-8cab-60d1a5ba97be\" (UID: \"28f3e4ff-5875-4e5d-8cab-60d1a5ba97be\") "
Oct 30 01:24:19.099932 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:19.099737    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-model-cache\") pod \"28f3e4ff-5875-4e5d-8cab-60d1a5ba97be\" (UID: \"28f3e4ff-5875-4e5d-8cab-60d1a5ba97be\") "
Oct 30 01:24:19.099932 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:19.099768    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-dshm\" (UniqueName: \"kubernetes.io/empty-dir/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-dshm\") pod \"28f3e4ff-5875-4e5d-8cab-60d1a5ba97be\" (UID: \"28f3e4ff-5875-4e5d-8cab-60d1a5ba97be\") "
Oct 30 01:24:19.099932 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:19.099915    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-kube-api-access-98tdz\" (UniqueName: \"kubernetes.io/projected/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-kube-api-access-98tdz\") pod \"28f3e4ff-5875-4e5d-8cab-60d1a5ba97be\" (UID: \"28f3e4ff-5875-4e5d-8cab-60d1a5ba97be\") "
Oct 30 01:24:19.100624 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:19.099962    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-home\" (UniqueName: \"kubernetes.io/empty-dir/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-home\") pod \"28f3e4ff-5875-4e5d-8cab-60d1a5ba97be\" (UID: \"28f3e4ff-5875-4e5d-8cab-60d1a5ba97be\") "
Oct 30 01:24:19.100624 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:19.100082    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-tmp-dir" pod "28f3e4ff-5875-4e5d-8cab-60d1a5ba97be" (UID: "28f3e4ff-5875-4e5d-8cab-60d1a5ba97be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:19.100624 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:19.100427    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-home" pod "28f3e4ff-5875-4e5d-8cab-60d1a5ba97be" (UID: "28f3e4ff-5875-4e5d-8cab-60d1a5ba97be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:19.100624 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:19.100483    4393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-tmp-dir\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:19.101048 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:19.100661    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-model-cache" pod "28f3e4ff-5875-4e5d-8cab-60d1a5ba97be" (UID: "28f3e4ff-5875-4e5d-8cab-60d1a5ba97be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:19.102671 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:19.102643    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-dshm" pod "28f3e4ff-5875-4e5d-8cab-60d1a5ba97be" (UID: "28f3e4ff-5875-4e5d-8cab-60d1a5ba97be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:19.102822 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:19.102712    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-kube-api-access-98tdz" pod "28f3e4ff-5875-4e5d-8cab-60d1a5ba97be" (UID: "28f3e4ff-5875-4e5d-8cab-60d1a5ba97be"). InnerVolumeSpecName "kube-api-access-98tdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:19.102899 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:19.102863    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-tls-certs" pod "28f3e4ff-5875-4e5d-8cab-60d1a5ba97be" (UID: "28f3e4ff-5875-4e5d-8cab-60d1a5ba97be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:19.202013 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:19.201973    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98tdz\" (UniqueName: \"kubernetes.io/projected/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-kube-api-access-98tdz\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:19.202013 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:19.202009    4393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-home\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:19.202013 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:19.202022    4393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-tls-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:19.202277 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:19.202035    4393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-model-cache\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:19.202277 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:19.202045    4393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be-dshm\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:19.955405 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:19.955364    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x" event={"ID":"28f3e4ff-5875-4e5d-8cab-60d1a5ba97be","Type":"ContainerDied","Data":"8f8404868ee9a1191e4cf1958c83c84148721ad6a744ebf81d18dc36982acd91"}
Oct 30 01:24:19.955950 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:19.955417    4393 scope.go:122] "RemoveContainer" containerID="2f989378070201471da6c09ec75f086bafd5930ca1f2003b959fc4103c2a7c3b"
Oct 30 01:24:19.955950 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:19.955378    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"
Oct 30 01:24:20.004700 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:20.004603    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"]
Oct 30 01:24:20.016072 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:20.016034    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-021d6d/distinct-2-w6-021d6d-kserve-6b7b66d485-h6j5x"]
Oct 30 01:24:21.272190 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:21.272149    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1a11cf-b489f44f9-lwz7b" podUID="1d8f254c-2610-4cb4-8e28-a59aea8a6de9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:24:21.725097 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:21.725059    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28f3e4ff-5875-4e5d-8cab-60d1a5ba97be" path="/var/lib/kubelet/pods/28f3e4ff-5875-4e5d-8cab-60d1a5ba97be/volumes"
Oct 30 01:24:23.882634 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:23.882590    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"]
Oct 30 01:24:23.883186 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:23.882871    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv" podUID="d70acab5-9907-49ce-84aa-0b41757e7ae0" containerName="istio-proxy" containerID="cri-o://5b5e79faf14232708e926597ebe7d3ca8f765d3925aacf89c685134c2ada02a4" gracePeriod=30
Oct 30 01:24:24.576548 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:24.576508    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw"]
Oct 30 01:24:24.576773 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:24.576728    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw" podUID="e55013a7-d78f-4e6e-bafe-cba27b9462b2" containerName="maas-api" containerID="cri-o://391ee6ab9e492a38f97c1a206aadccb26acb614f48a64f34625f148d87344bb0" gracePeriod=30
Oct 30 01:24:24.859271 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:24.859244    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw"
Oct 30 01:24:24.962888 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:24.962846    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e55013a7-d78f-4e6e-bafe-cba27b9462b2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e55013a7-d78f-4e6e-bafe-cba27b9462b2-openshift-service-ca\") pod \"e55013a7-d78f-4e6e-bafe-cba27b9462b2\" (UID: \"e55013a7-d78f-4e6e-bafe-cba27b9462b2\") "
Oct 30 01:24:24.963374 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:24.962906    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e55013a7-d78f-4e6e-bafe-cba27b9462b2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e55013a7-d78f-4e6e-bafe-cba27b9462b2-maas-api-tls\") pod \"e55013a7-d78f-4e6e-bafe-cba27b9462b2\" (UID: \"e55013a7-d78f-4e6e-bafe-cba27b9462b2\") "
Oct 30 01:24:24.963374 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:24.963089    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e55013a7-d78f-4e6e-bafe-cba27b9462b2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55013a7-d78f-4e6e-bafe-cba27b9462b2-metrics-certs\") pod \"e55013a7-d78f-4e6e-bafe-cba27b9462b2\" (UID: \"e55013a7-d78f-4e6e-bafe-cba27b9462b2\") "
Oct 30 01:24:24.963374 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:24.963190    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e55013a7-d78f-4e6e-bafe-cba27b9462b2-kube-api-access-wf884\" (UniqueName: \"kubernetes.io/projected/e55013a7-d78f-4e6e-bafe-cba27b9462b2-kube-api-access-wf884\") pod \"e55013a7-d78f-4e6e-bafe-cba27b9462b2\" (UID: \"e55013a7-d78f-4e6e-bafe-cba27b9462b2\") "
Oct 30 01:24:24.963374 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:24.963327    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e55013a7-d78f-4e6e-bafe-cba27b9462b2-openshift-service-ca" pod "e55013a7-d78f-4e6e-bafe-cba27b9462b2" (UID: "e55013a7-d78f-4e6e-bafe-cba27b9462b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:24.963885 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:24.963556    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e55013a7-d78f-4e6e-bafe-cba27b9462b2-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:24.965399 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:24.965372    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e55013a7-d78f-4e6e-bafe-cba27b9462b2-maas-api-tls" pod "e55013a7-d78f-4e6e-bafe-cba27b9462b2" (UID: "e55013a7-d78f-4e6e-bafe-cba27b9462b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:24.965399 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:24.965392    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e55013a7-d78f-4e6e-bafe-cba27b9462b2-metrics-certs" pod "e55013a7-d78f-4e6e-bafe-cba27b9462b2" (UID: "e55013a7-d78f-4e6e-bafe-cba27b9462b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:24.965566 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:24.965409    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e55013a7-d78f-4e6e-bafe-cba27b9462b2-kube-api-access-wf884" pod "e55013a7-d78f-4e6e-bafe-cba27b9462b2" (UID: "e55013a7-d78f-4e6e-bafe-cba27b9462b2"). InnerVolumeSpecName "kube-api-access-wf884". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:24.989305 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:24.989265    4393 generic.go:356] "Generic (PLEG): container finished" podID="e55013a7-d78f-4e6e-bafe-cba27b9462b2" containerID="391ee6ab9e492a38f97c1a206aadccb26acb614f48a64f34625f148d87344bb0" exitCode=0
Oct 30 01:24:24.989513 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:24.989356    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw" event={"ID":"e55013a7-d78f-4e6e-bafe-cba27b9462b2","Type":"ContainerDied","Data":"391ee6ab9e492a38f97c1a206aadccb26acb614f48a64f34625f148d87344bb0"}
Oct 30 01:24:24.989513 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:24.989388    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw"
Oct 30 01:24:24.989513 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:24.989402    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw" event={"ID":"e55013a7-d78f-4e6e-bafe-cba27b9462b2","Type":"ContainerDied","Data":"2463f05ab5870d19b8e9b6f2b3c3bc12be58fdbefa2fbd9ec43982adf2b97664"}
Oct 30 01:24:24.989513 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:24.989425    4393 scope.go:122] "RemoveContainer" containerID="391ee6ab9e492a38f97c1a206aadccb26acb614f48a64f34625f148d87344bb0"
Oct 30 01:24:25.000920 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:25.000896    4393 scope.go:122] "RemoveContainer" containerID="391ee6ab9e492a38f97c1a206aadccb26acb614f48a64f34625f148d87344bb0"
Oct 30 01:24:25.001265 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:24:25.001244    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"391ee6ab9e492a38f97c1a206aadccb26acb614f48a64f34625f148d87344bb0\": container with ID starting with 391ee6ab9e492a38f97c1a206aadccb26acb614f48a64f34625f148d87344bb0 not found: ID does not exist" containerID="391ee6ab9e492a38f97c1a206aadccb26acb614f48a64f34625f148d87344bb0"
Oct 30 01:24:25.001314 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:25.001275    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"391ee6ab9e492a38f97c1a206aadccb26acb614f48a64f34625f148d87344bb0"} err="failed to get container status \"391ee6ab9e492a38f97c1a206aadccb26acb614f48a64f34625f148d87344bb0\": rpc error: code = NotFound desc = could not find container \"391ee6ab9e492a38f97c1a206aadccb26acb614f48a64f34625f148d87344bb0\": container with ID starting with 391ee6ab9e492a38f97c1a206aadccb26acb614f48a64f34625f148d87344bb0 not found: ID does not exist"
Oct 30 01:24:25.044399 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:25.044352    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw"]
Oct 30 01:24:25.047995 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:25.047964    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-021d6d-7f5cc4bcbc-gplqw"]
Oct 30 01:24:25.065007 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:25.064971    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55013a7-d78f-4e6e-bafe-cba27b9462b2-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:25.065007 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:25.065004    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf884\" (UniqueName: \"kubernetes.io/projected/e55013a7-d78f-4e6e-bafe-cba27b9462b2-kube-api-access-wf884\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:25.065007 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:25.065016    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e55013a7-d78f-4e6e-bafe-cba27b9462b2-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:25.733759 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:25.733718    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e55013a7-d78f-4e6e-bafe-cba27b9462b2" path="/var/lib/kubelet/pods/e55013a7-d78f-4e6e-bafe-cba27b9462b2/volumes"
Oct 30 01:24:26.513309 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:24:26.513268    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:24:26.998712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:26.998684    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1a11cf-b489f44f9-lwz7b"
Oct 30 01:24:27.002974 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:27.002942    4393 generic.go:356] "Generic (PLEG): container finished" podID="1d8f254c-2610-4cb4-8e28-a59aea8a6de9" containerID="694a819e0ecd39bfb183417f157f995d61297b94a83e1dd353480a0c0a3a3246" exitCode=0
Oct 30 01:24:27.003146 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:27.002990    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1a11cf-b489f44f9-lwz7b"
Oct 30 01:24:27.003146 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:27.003028    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1a11cf-b489f44f9-lwz7b" event={"ID":"1d8f254c-2610-4cb4-8e28-a59aea8a6de9","Type":"ContainerDied","Data":"694a819e0ecd39bfb183417f157f995d61297b94a83e1dd353480a0c0a3a3246"}
Oct 30 01:24:27.003146 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:27.003099    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-1a11cf-b489f44f9-lwz7b" event={"ID":"1d8f254c-2610-4cb4-8e28-a59aea8a6de9","Type":"ContainerDied","Data":"a218151f50a8238c4c22bd7a5a51b7e4b3dacfe9549048e55831c5a8ef8a4d6e"}
Oct 30 01:24:27.003146 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:27.003132    4393 scope.go:122] "RemoveContainer" containerID="694a819e0ecd39bfb183417f157f995d61297b94a83e1dd353480a0c0a3a3246"
Oct 30 01:24:27.012891 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:27.012867    4393 scope.go:122] "RemoveContainer" containerID="694a819e0ecd39bfb183417f157f995d61297b94a83e1dd353480a0c0a3a3246"
Oct 30 01:24:27.013219 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:24:27.013185    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"694a819e0ecd39bfb183417f157f995d61297b94a83e1dd353480a0c0a3a3246\": container with ID starting with 694a819e0ecd39bfb183417f157f995d61297b94a83e1dd353480a0c0a3a3246 not found: ID does not exist" containerID="694a819e0ecd39bfb183417f157f995d61297b94a83e1dd353480a0c0a3a3246"
Oct 30 01:24:27.013350 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:27.013227    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"694a819e0ecd39bfb183417f157f995d61297b94a83e1dd353480a0c0a3a3246"} err="failed to get container status \"694a819e0ecd39bfb183417f157f995d61297b94a83e1dd353480a0c0a3a3246\": rpc error: code = NotFound desc = could not find container \"694a819e0ecd39bfb183417f157f995d61297b94a83e1dd353480a0c0a3a3246\": container with ID starting with 694a819e0ecd39bfb183417f157f995d61297b94a83e1dd353480a0c0a3a3246 not found: ID does not exist"
Oct 30 01:24:27.188400 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:27.188362    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d8f254c-2610-4cb4-8e28-a59aea8a6de9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d8f254c-2610-4cb4-8e28-a59aea8a6de9-plugins-config-volume\") pod \"1d8f254c-2610-4cb4-8e28-a59aea8a6de9\" (UID: \"1d8f254c-2610-4cb4-8e28-a59aea8a6de9\") "
Oct 30 01:24:27.188604 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:27.188448    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d8f254c-2610-4cb4-8e28-a59aea8a6de9-kube-api-access-pfgtk\" (UniqueName: \"kubernetes.io/projected/1d8f254c-2610-4cb4-8e28-a59aea8a6de9-kube-api-access-pfgtk\") pod \"1d8f254c-2610-4cb4-8e28-a59aea8a6de9\" (UID: \"1d8f254c-2610-4cb4-8e28-a59aea8a6de9\") "
Oct 30 01:24:27.188839 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:27.188771    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d8f254c-2610-4cb4-8e28-a59aea8a6de9-plugins-config-volume" pod "1d8f254c-2610-4cb4-8e28-a59aea8a6de9" (UID: "1d8f254c-2610-4cb4-8e28-a59aea8a6de9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:27.190866 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:27.190844    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d8f254c-2610-4cb4-8e28-a59aea8a6de9-kube-api-access-pfgtk" pod "1d8f254c-2610-4cb4-8e28-a59aea8a6de9" (UID: "1d8f254c-2610-4cb4-8e28-a59aea8a6de9"). InnerVolumeSpecName "kube-api-access-pfgtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:27.289677 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:27.289638    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d8f254c-2610-4cb4-8e28-a59aea8a6de9-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:27.289677 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:27.289674    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfgtk\" (UniqueName: \"kubernetes.io/projected/1d8f254c-2610-4cb4-8e28-a59aea8a6de9-kube-api-access-pfgtk\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:27.347231 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:27.347199    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-1a11cf-b489f44f9-lwz7b"]
Oct 30 01:24:27.358740 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:27.358705    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-1a11cf-b489f44f9-lwz7b"]
Oct 30 01:24:27.723715 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:27.723677    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d8f254c-2610-4cb4-8e28-a59aea8a6de9" path="/var/lib/kubelet/pods/1d8f254c-2610-4cb4-8e28-a59aea8a6de9/volumes"
Oct 30 01:24:29.018963 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:29.018929    4393 generic.go:356] "Generic (PLEG): container finished" podID="d70acab5-9907-49ce-84aa-0b41757e7ae0" containerID="5b5e79faf14232708e926597ebe7d3ca8f765d3925aacf89c685134c2ada02a4" exitCode=0
Oct 30 01:24:29.019337 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:29.019000    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv" event={"ID":"d70acab5-9907-49ce-84aa-0b41757e7ae0","Type":"ContainerDied","Data":"5b5e79faf14232708e926597ebe7d3ca8f765d3925aacf89c685134c2ada02a4"}
Oct 30 01:24:29.139947 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:29.139919    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"
Oct 30 01:24:29.205968 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:29.205862    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d70acab5-9907-49ce-84aa-0b41757e7ae0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70acab5-9907-49ce-84aa-0b41757e7ae0-workload-socket\") pod \"d70acab5-9907-49ce-84aa-0b41757e7ae0\" (UID: \"d70acab5-9907-49ce-84aa-0b41757e7ae0\") "
Oct 30 01:24:29.206169 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:29.206129    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70acab5-9907-49ce-84aa-0b41757e7ae0-workload-socket" pod "d70acab5-9907-49ce-84aa-0b41757e7ae0" (UID: "d70acab5-9907-49ce-84aa-0b41757e7ae0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:29.206169 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:29.206147    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d70acab5-9907-49ce-84aa-0b41757e7ae0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d70acab5-9907-49ce-84aa-0b41757e7ae0-workload-certs\") pod \"d70acab5-9907-49ce-84aa-0b41757e7ae0\" (UID: \"d70acab5-9907-49ce-84aa-0b41757e7ae0\") "
Oct 30 01:24:29.206301 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:29.206190    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d70acab5-9907-49ce-84aa-0b41757e7ae0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d70acab5-9907-49ce-84aa-0b41757e7ae0-istio-data\") pod \"d70acab5-9907-49ce-84aa-0b41757e7ae0\" (UID: \"d70acab5-9907-49ce-84aa-0b41757e7ae0\") "
Oct 30 01:24:29.206301 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:29.206224    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d70acab5-9907-49ce-84aa-0b41757e7ae0-istio-token\" (UniqueName: \"kubernetes.io/projected/d70acab5-9907-49ce-84aa-0b41757e7ae0-istio-token\") pod \"d70acab5-9907-49ce-84aa-0b41757e7ae0\" (UID: \"d70acab5-9907-49ce-84aa-0b41757e7ae0\") "
Oct 30 01:24:29.206301 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:29.206281    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d70acab5-9907-49ce-84aa-0b41757e7ae0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d70acab5-9907-49ce-84aa-0b41757e7ae0-istio-envoy\") pod \"d70acab5-9907-49ce-84aa-0b41757e7ae0\" (UID: \"d70acab5-9907-49ce-84aa-0b41757e7ae0\") "
Oct 30 01:24:29.206458 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:29.206331    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d70acab5-9907-49ce-84aa-0b41757e7ae0-kube-api-access-6gft2\" (UniqueName: \"kubernetes.io/projected/d70acab5-9907-49ce-84aa-0b41757e7ae0-kube-api-access-6gft2\") pod \"d70acab5-9907-49ce-84aa-0b41757e7ae0\" (UID: \"d70acab5-9907-49ce-84aa-0b41757e7ae0\") "
Oct 30 01:24:29.206458 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:29.206346    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70acab5-9907-49ce-84aa-0b41757e7ae0-workload-certs" pod "d70acab5-9907-49ce-84aa-0b41757e7ae0" (UID: "d70acab5-9907-49ce-84aa-0b41757e7ae0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:29.207758 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:29.206850    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d70acab5-9907-49ce-84aa-0b41757e7ae0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70acab5-9907-49ce-84aa-0b41757e7ae0-credential-socket\") pod \"d70acab5-9907-49ce-84aa-0b41757e7ae0\" (UID: \"d70acab5-9907-49ce-84aa-0b41757e7ae0\") "
Oct 30 01:24:29.207758 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:29.207157    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d70acab5-9907-49ce-84aa-0b41757e7ae0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d70acab5-9907-49ce-84aa-0b41757e7ae0-istio-podinfo\") pod \"d70acab5-9907-49ce-84aa-0b41757e7ae0\" (UID: \"d70acab5-9907-49ce-84aa-0b41757e7ae0\") "
Oct 30 01:24:29.207758 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:29.207188    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d70acab5-9907-49ce-84aa-0b41757e7ae0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d70acab5-9907-49ce-84aa-0b41757e7ae0-istiod-ca-cert\") pod \"d70acab5-9907-49ce-84aa-0b41757e7ae0\" (UID: \"d70acab5-9907-49ce-84aa-0b41757e7ae0\") "
Oct 30 01:24:29.207758 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:29.206952    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70acab5-9907-49ce-84aa-0b41757e7ae0-istio-data" pod "d70acab5-9907-49ce-84aa-0b41757e7ae0" (UID: "d70acab5-9907-49ce-84aa-0b41757e7ae0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:29.208641 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:29.208038    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70acab5-9907-49ce-84aa-0b41757e7ae0-credential-socket" pod "d70acab5-9907-49ce-84aa-0b41757e7ae0" (UID: "d70acab5-9907-49ce-84aa-0b41757e7ae0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:29.208924 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:29.208880    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d70acab5-9907-49ce-84aa-0b41757e7ae0-istiod-ca-cert" pod "d70acab5-9907-49ce-84aa-0b41757e7ae0" (UID: "d70acab5-9907-49ce-84aa-0b41757e7ae0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:29.209094 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:29.209065    4393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70acab5-9907-49ce-84aa-0b41757e7ae0-credential-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:29.209094 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:29.209088    4393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d70acab5-9907-49ce-84aa-0b41757e7ae0-istiod-ca-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:29.209223 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:29.209104    4393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70acab5-9907-49ce-84aa-0b41757e7ae0-workload-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:29.209223 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:29.209118    4393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d70acab5-9907-49ce-84aa-0b41757e7ae0-workload-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:29.209223 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:29.209132    4393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d70acab5-9907-49ce-84aa-0b41757e7ae0-istio-data\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:29.209410 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:29.209392    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d70acab5-9907-49ce-84aa-0b41757e7ae0-kube-api-access-6gft2" pod "d70acab5-9907-49ce-84aa-0b41757e7ae0" (UID: "d70acab5-9907-49ce-84aa-0b41757e7ae0"). InnerVolumeSpecName "kube-api-access-6gft2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:29.209688 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:29.209662    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d70acab5-9907-49ce-84aa-0b41757e7ae0-istio-token" pod "d70acab5-9907-49ce-84aa-0b41757e7ae0" (UID: "d70acab5-9907-49ce-84aa-0b41757e7ae0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:29.209833 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:29.209813    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70acab5-9907-49ce-84aa-0b41757e7ae0-istio-envoy" pod "d70acab5-9907-49ce-84aa-0b41757e7ae0" (UID: "d70acab5-9907-49ce-84aa-0b41757e7ae0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:29.210344 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:29.210326    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d70acab5-9907-49ce-84aa-0b41757e7ae0-istio-podinfo" pod "d70acab5-9907-49ce-84aa-0b41757e7ae0" (UID: "d70acab5-9907-49ce-84aa-0b41757e7ae0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:24:29.310511 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:29.310459    4393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d70acab5-9907-49ce-84aa-0b41757e7ae0-istio-envoy\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:29.310511 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:29.310510    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gft2\" (UniqueName: \"kubernetes.io/projected/d70acab5-9907-49ce-84aa-0b41757e7ae0-kube-api-access-6gft2\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:29.310511 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:29.310521    4393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d70acab5-9907-49ce-84aa-0b41757e7ae0-istio-podinfo\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:29.310511 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:29.310530    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d70acab5-9907-49ce-84aa-0b41757e7ae0-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:30.027324 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:30.027225    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv" event={"ID":"d70acab5-9907-49ce-84aa-0b41757e7ae0","Type":"ContainerDied","Data":"b9e419567cc1ac37f18ee9bfd6c604ce9b4af7f670b8a388b634288a2d7027c6"}
Oct 30 01:24:30.027324 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:30.027259    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"
Oct 30 01:24:30.027324 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:30.027299    4393 scope.go:122] "RemoveContainer" containerID="5b5e79faf14232708e926597ebe7d3ca8f765d3925aacf89c685134c2ada02a4"
Oct 30 01:24:30.090159 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:30.090126    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"]
Oct 30 01:24:30.093334 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:30.093308    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-1a11cf-openshift-default-748ddd859-zfzfv"]
Oct 30 01:24:31.723754 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:31.723717    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d70acab5-9907-49ce-84aa-0b41757e7ae0" path="/var/lib/kubelet/pods/d70acab5-9907-49ce-84aa-0b41757e7ae0/volumes"
Oct 30 01:24:32.100646 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:32.100540    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"]
Oct 30 01:24:32.100862 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:32.100823    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg" podUID="f42086fc-bfe1-4ded-be68-a54e07853fd2" containerName="istio-proxy" containerID="cri-o://629f5bd40027d52b5fe1e2de0fba03d2ee3bc356f32e72298cd3221d045be7dc" gracePeriod=30
Oct 30 01:24:37.355116 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.355084    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"
Oct 30 01:24:37.361538 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.361509    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b5b68ff8f-v68v9"]
Oct 30 01:24:37.361760 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.361696    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6b5b68ff8f-v68v9" podUID="7df9692e-c210-4300-8bcd-6fc93f7dd2fe" containerName="manager" containerID="cri-o://7085b5c7f1aae1d2d9b384c83716f964ad3edad3555862727f11807cdf66b24a" gracePeriod=30
Oct 30 01:24:37.389528 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.389505    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f42086fc-bfe1-4ded-be68-a54e07853fd2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f42086fc-bfe1-4ded-be68-a54e07853fd2-istio-data\") pod \"f42086fc-bfe1-4ded-be68-a54e07853fd2\" (UID: \"f42086fc-bfe1-4ded-be68-a54e07853fd2\") "
Oct 30 01:24:37.389644 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.389566    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f42086fc-bfe1-4ded-be68-a54e07853fd2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f42086fc-bfe1-4ded-be68-a54e07853fd2-workload-certs\") pod \"f42086fc-bfe1-4ded-be68-a54e07853fd2\" (UID: \"f42086fc-bfe1-4ded-be68-a54e07853fd2\") "
Oct 30 01:24:37.389644 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.389603    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f42086fc-bfe1-4ded-be68-a54e07853fd2-istio-token\" (UniqueName: \"kubernetes.io/projected/f42086fc-bfe1-4ded-be68-a54e07853fd2-istio-token\") pod \"f42086fc-bfe1-4ded-be68-a54e07853fd2\" (UID: \"f42086fc-bfe1-4ded-be68-a54e07853fd2\") "
Oct 30 01:24:37.389644 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.389638    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f42086fc-bfe1-4ded-be68-a54e07853fd2-kube-api-access-9m5bk\" (UniqueName: \"kubernetes.io/projected/f42086fc-bfe1-4ded-be68-a54e07853fd2-kube-api-access-9m5bk\") pod \"f42086fc-bfe1-4ded-be68-a54e07853fd2\" (UID: \"f42086fc-bfe1-4ded-be68-a54e07853fd2\") "
Oct 30 01:24:37.389827 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.389669    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f42086fc-bfe1-4ded-be68-a54e07853fd2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f42086fc-bfe1-4ded-be68-a54e07853fd2-istio-envoy\") pod \"f42086fc-bfe1-4ded-be68-a54e07853fd2\" (UID: \"f42086fc-bfe1-4ded-be68-a54e07853fd2\") "
Oct 30 01:24:37.389827 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.389764    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f42086fc-bfe1-4ded-be68-a54e07853fd2-istio-data" pod "f42086fc-bfe1-4ded-be68-a54e07853fd2" (UID: "f42086fc-bfe1-4ded-be68-a54e07853fd2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:37.389948 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.389835    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f42086fc-bfe1-4ded-be68-a54e07853fd2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f42086fc-bfe1-4ded-be68-a54e07853fd2-workload-socket\") pod \"f42086fc-bfe1-4ded-be68-a54e07853fd2\" (UID: \"f42086fc-bfe1-4ded-be68-a54e07853fd2\") "
Oct 30 01:24:37.389948 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.389851    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f42086fc-bfe1-4ded-be68-a54e07853fd2-workload-certs" pod "f42086fc-bfe1-4ded-be68-a54e07853fd2" (UID: "f42086fc-bfe1-4ded-be68-a54e07853fd2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:37.389948 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.389916    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f42086fc-bfe1-4ded-be68-a54e07853fd2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f42086fc-bfe1-4ded-be68-a54e07853fd2-istiod-ca-cert\") pod \"f42086fc-bfe1-4ded-be68-a54e07853fd2\" (UID: \"f42086fc-bfe1-4ded-be68-a54e07853fd2\") "
Oct 30 01:24:37.389948 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.389939    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f42086fc-bfe1-4ded-be68-a54e07853fd2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f42086fc-bfe1-4ded-be68-a54e07853fd2-istio-podinfo\") pod \"f42086fc-bfe1-4ded-be68-a54e07853fd2\" (UID: \"f42086fc-bfe1-4ded-be68-a54e07853fd2\") "
Oct 30 01:24:37.390158 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.389990    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f42086fc-bfe1-4ded-be68-a54e07853fd2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f42086fc-bfe1-4ded-be68-a54e07853fd2-credential-socket\") pod \"f42086fc-bfe1-4ded-be68-a54e07853fd2\" (UID: \"f42086fc-bfe1-4ded-be68-a54e07853fd2\") "
Oct 30 01:24:37.390158 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.390032    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f42086fc-bfe1-4ded-be68-a54e07853fd2-workload-socket" pod "f42086fc-bfe1-4ded-be68-a54e07853fd2" (UID: "f42086fc-bfe1-4ded-be68-a54e07853fd2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:37.390408 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.390341    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f42086fc-bfe1-4ded-be68-a54e07853fd2-credential-socket" pod "f42086fc-bfe1-4ded-be68-a54e07853fd2" (UID: "f42086fc-bfe1-4ded-be68-a54e07853fd2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:37.390408 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.390352    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f42086fc-bfe1-4ded-be68-a54e07853fd2-istiod-ca-cert" pod "f42086fc-bfe1-4ded-be68-a54e07853fd2" (UID: "f42086fc-bfe1-4ded-be68-a54e07853fd2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:24:37.390408 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.390403    4393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f42086fc-bfe1-4ded-be68-a54e07853fd2-istio-data\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:37.390857 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.390423    4393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f42086fc-bfe1-4ded-be68-a54e07853fd2-workload-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:37.390857 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.390441    4393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f42086fc-bfe1-4ded-be68-a54e07853fd2-workload-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:37.392185 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.392158    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f42086fc-bfe1-4ded-be68-a54e07853fd2-istio-podinfo" pod "f42086fc-bfe1-4ded-be68-a54e07853fd2" (UID: "f42086fc-bfe1-4ded-be68-a54e07853fd2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:24:37.392876 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.392601    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f42086fc-bfe1-4ded-be68-a54e07853fd2-kube-api-access-9m5bk" pod "f42086fc-bfe1-4ded-be68-a54e07853fd2" (UID: "f42086fc-bfe1-4ded-be68-a54e07853fd2"). InnerVolumeSpecName "kube-api-access-9m5bk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:37.392876 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.392641    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f42086fc-bfe1-4ded-be68-a54e07853fd2-istio-token" pod "f42086fc-bfe1-4ded-be68-a54e07853fd2" (UID: "f42086fc-bfe1-4ded-be68-a54e07853fd2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:37.392876 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.392745    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f42086fc-bfe1-4ded-be68-a54e07853fd2-istio-envoy" pod "f42086fc-bfe1-4ded-be68-a54e07853fd2" (UID: "f42086fc-bfe1-4ded-be68-a54e07853fd2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:24:37.491646 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.491610    4393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f42086fc-bfe1-4ded-be68-a54e07853fd2-credential-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:37.491771 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.491649    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f42086fc-bfe1-4ded-be68-a54e07853fd2-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:37.491771 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.491692    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m5bk\" (UniqueName: \"kubernetes.io/projected/f42086fc-bfe1-4ded-be68-a54e07853fd2-kube-api-access-9m5bk\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:37.491771 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.491706    4393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f42086fc-bfe1-4ded-be68-a54e07853fd2-istio-envoy\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:37.491771 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.491720    4393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f42086fc-bfe1-4ded-be68-a54e07853fd2-istiod-ca-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:37.491771 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.491732    4393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f42086fc-bfe1-4ded-be68-a54e07853fd2-istio-podinfo\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:37.622676 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.622651    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b5b68ff8f-v68v9"
Oct 30 01:24:37.694001 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.693908    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7df9692e-c210-4300-8bcd-6fc93f7dd2fe-kube-api-access-mwcqc\" (UniqueName: \"kubernetes.io/projected/7df9692e-c210-4300-8bcd-6fc93f7dd2fe-kube-api-access-mwcqc\") pod \"7df9692e-c210-4300-8bcd-6fc93f7dd2fe\" (UID: \"7df9692e-c210-4300-8bcd-6fc93f7dd2fe\") "
Oct 30 01:24:37.694171 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.694030    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7df9692e-c210-4300-8bcd-6fc93f7dd2fe-webhook-cert\" (UniqueName: \"kubernetes.io/secret/7df9692e-c210-4300-8bcd-6fc93f7dd2fe-webhook-cert\") pod \"7df9692e-c210-4300-8bcd-6fc93f7dd2fe\" (UID: \"7df9692e-c210-4300-8bcd-6fc93f7dd2fe\") "
Oct 30 01:24:37.694171 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.694123    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7df9692e-c210-4300-8bcd-6fc93f7dd2fe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7df9692e-c210-4300-8bcd-6fc93f7dd2fe-metrics-certs\") pod \"7df9692e-c210-4300-8bcd-6fc93f7dd2fe\" (UID: \"7df9692e-c210-4300-8bcd-6fc93f7dd2fe\") "
Oct 30 01:24:37.696369 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.696332    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7df9692e-c210-4300-8bcd-6fc93f7dd2fe-webhook-cert" pod "7df9692e-c210-4300-8bcd-6fc93f7dd2fe" (UID: "7df9692e-c210-4300-8bcd-6fc93f7dd2fe"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:37.696369 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.696360    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7df9692e-c210-4300-8bcd-6fc93f7dd2fe-kube-api-access-mwcqc" pod "7df9692e-c210-4300-8bcd-6fc93f7dd2fe" (UID: "7df9692e-c210-4300-8bcd-6fc93f7dd2fe"). InnerVolumeSpecName "kube-api-access-mwcqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:24:37.696620 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.696597    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7df9692e-c210-4300-8bcd-6fc93f7dd2fe-metrics-certs" pod "7df9692e-c210-4300-8bcd-6fc93f7dd2fe" (UID: "7df9692e-c210-4300-8bcd-6fc93f7dd2fe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:24:37.795221 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.795188    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7df9692e-c210-4300-8bcd-6fc93f7dd2fe-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:37.795221 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.795218    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwcqc\" (UniqueName: \"kubernetes.io/projected/7df9692e-c210-4300-8bcd-6fc93f7dd2fe-kube-api-access-mwcqc\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:37.795221 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:37.795230    4393 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7df9692e-c210-4300-8bcd-6fc93f7dd2fe-webhook-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:24:38.072967 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:38.072876    4393 generic.go:356] "Generic (PLEG): container finished" podID="f42086fc-bfe1-4ded-be68-a54e07853fd2" containerID="629f5bd40027d52b5fe1e2de0fba03d2ee3bc356f32e72298cd3221d045be7dc" exitCode=0
Oct 30 01:24:38.072967 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:38.072937    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"
Oct 30 01:24:38.072967 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:38.072944    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg" event={"ID":"f42086fc-bfe1-4ded-be68-a54e07853fd2","Type":"ContainerDied","Data":"629f5bd40027d52b5fe1e2de0fba03d2ee3bc356f32e72298cd3221d045be7dc"}
Oct 30 01:24:38.073277 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:38.072997    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg" event={"ID":"f42086fc-bfe1-4ded-be68-a54e07853fd2","Type":"ContainerDied","Data":"f7e50d68227dc30f74e1fec9c313bdac77199f415b6c25794858898bdd7eb89c"}
Oct 30 01:24:38.073277 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:38.073026    4393 scope.go:122] "RemoveContainer" containerID="629f5bd40027d52b5fe1e2de0fba03d2ee3bc356f32e72298cd3221d045be7dc"
Oct 30 01:24:38.074405 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:38.074383    4393 generic.go:356] "Generic (PLEG): container finished" podID="7df9692e-c210-4300-8bcd-6fc93f7dd2fe" containerID="7085b5c7f1aae1d2d9b384c83716f964ad3edad3555862727f11807cdf66b24a" exitCode=0
Oct 30 01:24:38.074496 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:38.074424    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b5b68ff8f-v68v9"
Oct 30 01:24:38.074549 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:38.074426    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b5b68ff8f-v68v9" event={"ID":"7df9692e-c210-4300-8bcd-6fc93f7dd2fe","Type":"ContainerDied","Data":"7085b5c7f1aae1d2d9b384c83716f964ad3edad3555862727f11807cdf66b24a"}
Oct 30 01:24:38.074618 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:38.074562    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b5b68ff8f-v68v9" event={"ID":"7df9692e-c210-4300-8bcd-6fc93f7dd2fe","Type":"ContainerDied","Data":"0bfab7b2c6283ae2e54875f06e764b53f117c58605d3e0e62f82e5432b7129d9"}
Oct 30 01:24:38.082919 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:38.082897    4393 scope.go:122] "RemoveContainer" containerID="629f5bd40027d52b5fe1e2de0fba03d2ee3bc356f32e72298cd3221d045be7dc"
Oct 30 01:24:38.083213 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:24:38.083194    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"629f5bd40027d52b5fe1e2de0fba03d2ee3bc356f32e72298cd3221d045be7dc\": container with ID starting with 629f5bd40027d52b5fe1e2de0fba03d2ee3bc356f32e72298cd3221d045be7dc not found: ID does not exist" containerID="629f5bd40027d52b5fe1e2de0fba03d2ee3bc356f32e72298cd3221d045be7dc"
Oct 30 01:24:38.083293 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:38.083221    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"629f5bd40027d52b5fe1e2de0fba03d2ee3bc356f32e72298cd3221d045be7dc"} err="failed to get container status \"629f5bd40027d52b5fe1e2de0fba03d2ee3bc356f32e72298cd3221d045be7dc\": rpc error: code = NotFound desc = could not find container \"629f5bd40027d52b5fe1e2de0fba03d2ee3bc356f32e72298cd3221d045be7dc\": container with ID starting with 629f5bd40027d52b5fe1e2de0fba03d2ee3bc356f32e72298cd3221d045be7dc not found: ID does not exist"
Oct 30 01:24:38.083293 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:38.083236    4393 scope.go:122] "RemoveContainer" containerID="7085b5c7f1aae1d2d9b384c83716f964ad3edad3555862727f11807cdf66b24a"
Oct 30 01:24:38.093556 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:38.093535    4393 scope.go:122] "RemoveContainer" containerID="7085b5c7f1aae1d2d9b384c83716f964ad3edad3555862727f11807cdf66b24a"
Oct 30 01:24:38.093831 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:24:38.093812    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7085b5c7f1aae1d2d9b384c83716f964ad3edad3555862727f11807cdf66b24a\": container with ID starting with 7085b5c7f1aae1d2d9b384c83716f964ad3edad3555862727f11807cdf66b24a not found: ID does not exist" containerID="7085b5c7f1aae1d2d9b384c83716f964ad3edad3555862727f11807cdf66b24a"
Oct 30 01:24:38.093894 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:38.093838    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7085b5c7f1aae1d2d9b384c83716f964ad3edad3555862727f11807cdf66b24a"} err="failed to get container status \"7085b5c7f1aae1d2d9b384c83716f964ad3edad3555862727f11807cdf66b24a\": rpc error: code = NotFound desc = could not find container \"7085b5c7f1aae1d2d9b384c83716f964ad3edad3555862727f11807cdf66b24a\": container with ID starting with 7085b5c7f1aae1d2d9b384c83716f964ad3edad3555862727f11807cdf66b24a not found: ID does not exist"
Oct 30 01:24:38.124671 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:38.124637    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"]
Oct 30 01:24:38.135177 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:38.135149    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-021d6d-openshift-default-65c58b6947-mclhg"]
Oct 30 01:24:38.163885 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:38.163855    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b5b68ff8f-v68v9"]
Oct 30 01:24:38.164210 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:38.164188    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6b5b68ff8f-v68v9"]
Oct 30 01:24:39.103205 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103171    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6b5b68ff8f-9kmqm"]
Oct 30 01:24:39.103645 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103583    4393 cpu_manager.go:414] "containerMap: removing container" podUID="de3f97cd-2787-4b28-a883-18c1f36ca325" containerName="payload-processing"
Oct 30 01:24:39.103645 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103594    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de3f97cd-2787-4b28-a883-18c1f36ca325" containerName="payload-processing"
Oct 30 01:24:39.103645 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103609    4393 cpu_manager.go:414] "containerMap: removing container" podUID="28f3e4ff-5875-4e5d-8cab-60d1a5ba97be" containerName="main"
Oct 30 01:24:39.103645 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103615    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28f3e4ff-5875-4e5d-8cab-60d1a5ba97be" containerName="main"
Oct 30 01:24:39.103645 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103625    4393 cpu_manager.go:414] "containerMap: removing container" podUID="f42086fc-bfe1-4ded-be68-a54e07853fd2" containerName="istio-proxy"
Oct 30 01:24:39.103645 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103629    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f42086fc-bfe1-4ded-be68-a54e07853fd2" containerName="istio-proxy"
Oct 30 01:24:39.103645 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103636    4393 cpu_manager.go:414] "containerMap: removing container" podUID="7416ca5a-6ff8-4937-84cf-15a156c523eb" containerName="maas-api"
Oct 30 01:24:39.103645 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103642    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7416ca5a-6ff8-4937-84cf-15a156c523eb" containerName="maas-api"
Oct 30 01:24:39.103645 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103648    4393 cpu_manager.go:414] "containerMap: removing container" podUID="b31f25ea-0447-4caf-9237-d59594938f72" containerName="istio-proxy"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103653    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b31f25ea-0447-4caf-9237-d59594938f72" containerName="istio-proxy"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103659    4393 cpu_manager.go:414] "containerMap: removing container" podUID="07623c58-4919-43fe-ad87-6e47c6af9596" containerName="istio-proxy"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103663    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07623c58-4919-43fe-ad87-6e47c6af9596" containerName="istio-proxy"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103671    4393 cpu_manager.go:414] "containerMap: removing container" podUID="38092938-4c3d-41ae-b7dd-25ddd1869d7d" containerName="maas-api"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103675    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38092938-4c3d-41ae-b7dd-25ddd1869d7d" containerName="maas-api"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103682    4393 cpu_manager.go:414] "containerMap: removing container" podUID="a674fcd8-3f60-4f70-b199-a2ad09d74fb2" containerName="payload-processing"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103686    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a674fcd8-3f60-4f70-b199-a2ad09d74fb2" containerName="payload-processing"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103693    4393 cpu_manager.go:414] "containerMap: removing container" podUID="0fa2bddd-6b15-47a8-95cc-8f952594e284" containerName="payload-pre-processing"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103697    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fa2bddd-6b15-47a8-95cc-8f952594e284" containerName="payload-pre-processing"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103703    4393 cpu_manager.go:414] "containerMap: removing container" podUID="1e044c11-fa02-43cd-af55-42b87c0b6789" containerName="payload-pre-processing"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103708    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e044c11-fa02-43cd-af55-42b87c0b6789" containerName="payload-pre-processing"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103716    4393 cpu_manager.go:414] "containerMap: removing container" podUID="a88ea59f-24dc-42df-8bd5-ffffdb5f36f2" containerName="maas-api"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103721    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a88ea59f-24dc-42df-8bd5-ffffdb5f36f2" containerName="maas-api"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103728    4393 cpu_manager.go:414] "containerMap: removing container" podUID="e1ec248b-a567-4fb4-a415-ac1235608942" containerName="payload-processing"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103732    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1ec248b-a567-4fb4-a415-ac1235608942" containerName="payload-processing"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103739    4393 cpu_manager.go:414] "containerMap: removing container" podUID="691e02fc-7fd6-454b-a78e-31e3c34175c3" containerName="payload-pre-processing"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103744    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="691e02fc-7fd6-454b-a78e-31e3c34175c3" containerName="payload-pre-processing"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103750    4393 cpu_manager.go:414] "containerMap: removing container" podUID="77896a6a-93c0-4f84-91ff-676e591f4453" containerName="maas-api"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103754    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77896a6a-93c0-4f84-91ff-676e591f4453" containerName="maas-api"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103759    4393 cpu_manager.go:414] "containerMap: removing container" podUID="d5355e22-6596-4c63-a492-fe8b578f04b5" containerName="istio-proxy"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103763    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5355e22-6596-4c63-a492-fe8b578f04b5" containerName="istio-proxy"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103769    4393 cpu_manager.go:414] "containerMap: removing container" podUID="c026d820-c4ea-4d97-97b7-18757b273b8e" containerName="istio-proxy"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103773    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c026d820-c4ea-4d97-97b7-18757b273b8e" containerName="istio-proxy"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103794    4393 cpu_manager.go:414] "containerMap: removing container" podUID="63d6236e-0643-4747-b66d-bbdee696584d" containerName="maas-api"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103798    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63d6236e-0643-4747-b66d-bbdee696584d" containerName="maas-api"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103807    4393 cpu_manager.go:414] "containerMap: removing container" podUID="85f8ce06-4e78-43be-9c07-05c201eb70e8" containerName="maas-api"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103811    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f8ce06-4e78-43be-9c07-05c201eb70e8" containerName="maas-api"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103817    4393 cpu_manager.go:414] "containerMap: removing container" podUID="3847bedb-934b-4dac-a5c1-7e5d593ba2ec" containerName="payload-processing"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103823    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3847bedb-934b-4dac-a5c1-7e5d593ba2ec" containerName="payload-processing"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103829    4393 cpu_manager.go:414] "containerMap: removing container" podUID="4014c968-b2d9-4189-b8fb-18de72da909b" containerName="maas-api"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103833    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4014c968-b2d9-4189-b8fb-18de72da909b" containerName="maas-api"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103839    4393 cpu_manager.go:414] "containerMap: removing container" podUID="6bfbd1a2-2a38-43ed-abe0-c1a719454e10" containerName="payload-pre-processing"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103843    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bfbd1a2-2a38-43ed-abe0-c1a719454e10" containerName="payload-pre-processing"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103850    4393 cpu_manager.go:414] "containerMap: removing container" podUID="1d8f254c-2610-4cb4-8e28-a59aea8a6de9" containerName="payload-pre-processing"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103855    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d8f254c-2610-4cb4-8e28-a59aea8a6de9" containerName="payload-pre-processing"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103861    4393 cpu_manager.go:414] "containerMap: removing container" podUID="56a3d613-1b6a-4e50-b5e3-517002e2c5a8" containerName="maas-api"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103865    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a3d613-1b6a-4e50-b5e3-517002e2c5a8" containerName="maas-api"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103876    4393 cpu_manager.go:414] "containerMap: removing container" podUID="be58de6c-a4f3-40e5-9270-689a5eee91f1" containerName="payload-pre-processing"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103881    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be58de6c-a4f3-40e5-9270-689a5eee91f1" containerName="payload-pre-processing"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103887    4393 cpu_manager.go:414] "containerMap: removing container" podUID="9f083e40-d31b-4313-8459-f5ab9ac9d743" containerName="istio-proxy"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103891    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f083e40-d31b-4313-8459-f5ab9ac9d743" containerName="istio-proxy"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103897    4393 cpu_manager.go:414] "containerMap: removing container" podUID="4debacaf-9ed1-4e85-a8f3-4f52a76ef705" containerName="payload-pre-processing"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103901    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4debacaf-9ed1-4e85-a8f3-4f52a76ef705" containerName="payload-pre-processing"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103909    4393 cpu_manager.go:414] "containerMap: removing container" podUID="08c5615b-d854-4dd7-843e-9764dab5a127" containerName="maas-api"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103915    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08c5615b-d854-4dd7-843e-9764dab5a127" containerName="maas-api"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103923    4393 cpu_manager.go:414] "containerMap: removing container" podUID="e55013a7-d78f-4e6e-bafe-cba27b9462b2" containerName="maas-api"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103928    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e55013a7-d78f-4e6e-bafe-cba27b9462b2" containerName="maas-api"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103933    4393 cpu_manager.go:414] "containerMap: removing container" podUID="d70acab5-9907-49ce-84aa-0b41757e7ae0" containerName="istio-proxy"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103937    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d70acab5-9907-49ce-84aa-0b41757e7ae0" containerName="istio-proxy"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103944    4393 cpu_manager.go:414] "containerMap: removing container" podUID="6da14397-0216-4ada-83bf-72addba282e5" containerName="maas-api"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103948    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6da14397-0216-4ada-83bf-72addba282e5" containerName="maas-api"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103954    4393 cpu_manager.go:414] "containerMap: removing container" podUID="2526f9aa-9b27-4a09-b321-ec07b6b0131e" containerName="maas-api"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103958    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2526f9aa-9b27-4a09-b321-ec07b6b0131e" containerName="maas-api"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103965    4393 cpu_manager.go:414] "containerMap: removing container" podUID="fb10b912-9abe-4547-83a8-aa63ddb14599" containerName="maas-api"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103969    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb10b912-9abe-4547-83a8-aa63ddb14599" containerName="maas-api"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103975    4393 cpu_manager.go:414] "containerMap: removing container" podUID="40a26137-dc63-40e1-89cc-2a0c70d6f52e" containerName="istio-proxy"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103981    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40a26137-dc63-40e1-89cc-2a0c70d6f52e" containerName="istio-proxy"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103992    4393 cpu_manager.go:414] "containerMap: removing container" podUID="7028bbc0-dd7e-4525-a1db-8bf264d8ad58" containerName="maas-api"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.103999    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7028bbc0-dd7e-4525-a1db-8bf264d8ad58" containerName="maas-api"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104013    4393 cpu_manager.go:414] "containerMap: removing container" podUID="20350585-8ccb-4f19-a634-c2cc4031b330" containerName="payload-processing"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104021    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20350585-8ccb-4f19-a634-c2cc4031b330" containerName="payload-processing"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104029    4393 cpu_manager.go:414] "containerMap: removing container" podUID="dd52771e-289e-4bd7-952e-eaddd165eb58" containerName="payload-processing"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104037    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd52771e-289e-4bd7-952e-eaddd165eb58" containerName="payload-processing"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104048    4393 cpu_manager.go:414] "containerMap: removing container" podUID="4ea0d2c2-401c-4873-ac44-75481cdfd848" containerName="maas-api"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104055    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea0d2c2-401c-4873-ac44-75481cdfd848" containerName="maas-api"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104068    4393 cpu_manager.go:414] "containerMap: removing container" podUID="7df9692e-c210-4300-8bcd-6fc93f7dd2fe" containerName="manager"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104075    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7df9692e-c210-4300-8bcd-6fc93f7dd2fe" containerName="manager"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104085    4393 cpu_manager.go:414] "containerMap: removing container" podUID="473e63de-a201-41a6-b927-eee7643de4e1" containerName="maas-api"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104092    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="473e63de-a201-41a6-b927-eee7643de4e1" containerName="maas-api"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104104    4393 cpu_manager.go:414] "containerMap: removing container" podUID="1e34134b-9305-4376-b55b-c08c72cfbedd" containerName="istio-proxy"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104111    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e34134b-9305-4376-b55b-c08c72cfbedd" containerName="istio-proxy"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104121    4393 cpu_manager.go:414] "containerMap: removing container" podUID="46b01388-7da7-4e63-95f3-afc811bed013" containerName="main"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104128    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46b01388-7da7-4e63-95f3-afc811bed013" containerName="main"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104137    4393 cpu_manager.go:414] "containerMap: removing container" podUID="2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd" containerName="payload-processing"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104144    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd" containerName="payload-processing"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104152    4393 cpu_manager.go:414] "containerMap: removing container" podUID="6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4" containerName="maas-api"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104158    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4" containerName="maas-api"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104172    4393 cpu_manager.go:414] "containerMap: removing container" podUID="44688868-ab51-4384-9887-b1d61810eca3" containerName="payload-pre-processing"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104179    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44688868-ab51-4384-9887-b1d61810eca3" containerName="payload-pre-processing"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104192    4393 cpu_manager.go:414] "containerMap: removing container" podUID="a002f857-7a6e-4fb6-a05c-bce70bac9342" containerName="maas-api"
Oct 30 01:24:39.104222 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104198    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a002f857-7a6e-4fb6-a05c-bce70bac9342" containerName="maas-api"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104314    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="de3f97cd-2787-4b28-a883-18c1f36ca325" containerName="payload-processing"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104328    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="28f3e4ff-5875-4e5d-8cab-60d1a5ba97be" containerName="main"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104336    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="f42086fc-bfe1-4ded-be68-a54e07853fd2" containerName="istio-proxy"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104343    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="7416ca5a-6ff8-4937-84cf-15a156c523eb" containerName="maas-api"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104348    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="b31f25ea-0447-4caf-9237-d59594938f72" containerName="istio-proxy"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104358    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="07623c58-4919-43fe-ad87-6e47c6af9596" containerName="istio-proxy"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104368    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="38092938-4c3d-41ae-b7dd-25ddd1869d7d" containerName="maas-api"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104376    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="a674fcd8-3f60-4f70-b199-a2ad09d74fb2" containerName="payload-processing"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104382    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fa2bddd-6b15-47a8-95cc-8f952594e284" containerName="payload-pre-processing"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104388    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e044c11-fa02-43cd-af55-42b87c0b6789" containerName="payload-pre-processing"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104393    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="a88ea59f-24dc-42df-8bd5-ffffdb5f36f2" containerName="maas-api"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104398    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1ec248b-a567-4fb4-a415-ac1235608942" containerName="payload-processing"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104404    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="691e02fc-7fd6-454b-a78e-31e3c34175c3" containerName="payload-pre-processing"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104409    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="77896a6a-93c0-4f84-91ff-676e591f4453" containerName="maas-api"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104413    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5355e22-6596-4c63-a492-fe8b578f04b5" containerName="istio-proxy"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104419    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="c026d820-c4ea-4d97-97b7-18757b273b8e" containerName="istio-proxy"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104424    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="63d6236e-0643-4747-b66d-bbdee696584d" containerName="maas-api"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104431    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f8ce06-4e78-43be-9c07-05c201eb70e8" containerName="maas-api"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104435    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="3847bedb-934b-4dac-a5c1-7e5d593ba2ec" containerName="payload-processing"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104441    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="4014c968-b2d9-4189-b8fb-18de72da909b" containerName="maas-api"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104445    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bfbd1a2-2a38-43ed-abe0-c1a719454e10" containerName="payload-pre-processing"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104451    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d8f254c-2610-4cb4-8e28-a59aea8a6de9" containerName="payload-pre-processing"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104457    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="56a3d613-1b6a-4e50-b5e3-517002e2c5a8" containerName="maas-api"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104465    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="be58de6c-a4f3-40e5-9270-689a5eee91f1" containerName="payload-pre-processing"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104470    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f083e40-d31b-4313-8459-f5ab9ac9d743" containerName="istio-proxy"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104477    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="4debacaf-9ed1-4e85-a8f3-4f52a76ef705" containerName="payload-pre-processing"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104484    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="08c5615b-d854-4dd7-843e-9764dab5a127" containerName="maas-api"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104491    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="e55013a7-d78f-4e6e-bafe-cba27b9462b2" containerName="maas-api"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104496    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="d70acab5-9907-49ce-84aa-0b41757e7ae0" containerName="istio-proxy"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104502    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="6da14397-0216-4ada-83bf-72addba282e5" containerName="maas-api"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104507    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="2526f9aa-9b27-4a09-b321-ec07b6b0131e" containerName="maas-api"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104511    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb10b912-9abe-4547-83a8-aa63ddb14599" containerName="maas-api"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104517    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="40a26137-dc63-40e1-89cc-2a0c70d6f52e" containerName="istio-proxy"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104523    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="7028bbc0-dd7e-4525-a1db-8bf264d8ad58" containerName="maas-api"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104529    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="20350585-8ccb-4f19-a634-c2cc4031b330" containerName="payload-processing"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104534    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd52771e-289e-4bd7-952e-eaddd165eb58" containerName="payload-processing"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104539    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea0d2c2-401c-4873-ac44-75481cdfd848" containerName="maas-api"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104544    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="7df9692e-c210-4300-8bcd-6fc93f7dd2fe" containerName="manager"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104550    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="473e63de-a201-41a6-b927-eee7643de4e1" containerName="maas-api"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104556    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e34134b-9305-4376-b55b-c08c72cfbedd" containerName="istio-proxy"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104562    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="46b01388-7da7-4e63-95f3-afc811bed013" containerName="main"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104567    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fd09a90-3f1e-4fc9-9ff8-fd8bab9a22cd" containerName="payload-processing"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104572    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="6634f2ef-e6cc-4dc2-bc18-55cf33bc9cd4" containerName="maas-api"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104577    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="44688868-ab51-4384-9887-b1d61810eca3" containerName="payload-pre-processing"
Oct 30 01:24:39.106712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.104583    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="a002f857-7a6e-4fb6-a05c-bce70bac9342" containerName="maas-api"
Oct 30 01:24:39.109209 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.109191    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b5b68ff8f-9kmqm"
Oct 30 01:24:39.123297 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.123252    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 01:24:39.134980 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.134945    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 01:24:39.134980 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.134971    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vzmlw\""
Oct 30 01:24:39.135248 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.134946    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 01:24:39.137470 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.137443    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b5b68ff8f-9kmqm"]
Oct 30 01:24:39.208928 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.208884    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/542d6180-0507-48b6-bd4e-a79b055b1195-metrics-certs\") pod \"maas-controller-6b5b68ff8f-9kmqm\" (UID: \"542d6180-0507-48b6-bd4e-a79b055b1195\") " pod="opendatahub/maas-controller-6b5b68ff8f-9kmqm"
Oct 30 01:24:39.209105 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.209045    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/542d6180-0507-48b6-bd4e-a79b055b1195-webhook-cert\") pod \"maas-controller-6b5b68ff8f-9kmqm\" (UID: \"542d6180-0507-48b6-bd4e-a79b055b1195\") " pod="opendatahub/maas-controller-6b5b68ff8f-9kmqm"
Oct 30 01:24:39.209105 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.209076    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h9f2\" (UniqueName: \"kubernetes.io/projected/542d6180-0507-48b6-bd4e-a79b055b1195-kube-api-access-6h9f2\") pod \"maas-controller-6b5b68ff8f-9kmqm\" (UID: \"542d6180-0507-48b6-bd4e-a79b055b1195\") " pod="opendatahub/maas-controller-6b5b68ff8f-9kmqm"
Oct 30 01:24:39.309977 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.309930    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h9f2\" (UniqueName: \"kubernetes.io/projected/542d6180-0507-48b6-bd4e-a79b055b1195-kube-api-access-6h9f2\") pod \"maas-controller-6b5b68ff8f-9kmqm\" (UID: \"542d6180-0507-48b6-bd4e-a79b055b1195\") " pod="opendatahub/maas-controller-6b5b68ff8f-9kmqm"
Oct 30 01:24:39.310183 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.310005    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/542d6180-0507-48b6-bd4e-a79b055b1195-metrics-certs\") pod \"maas-controller-6b5b68ff8f-9kmqm\" (UID: \"542d6180-0507-48b6-bd4e-a79b055b1195\") " pod="opendatahub/maas-controller-6b5b68ff8f-9kmqm"
Oct 30 01:24:39.310183 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.310098    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/542d6180-0507-48b6-bd4e-a79b055b1195-webhook-cert\") pod \"maas-controller-6b5b68ff8f-9kmqm\" (UID: \"542d6180-0507-48b6-bd4e-a79b055b1195\") " pod="opendatahub/maas-controller-6b5b68ff8f-9kmqm"
Oct 30 01:24:39.312754 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.312730    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/542d6180-0507-48b6-bd4e-a79b055b1195-metrics-certs\") pod \"maas-controller-6b5b68ff8f-9kmqm\" (UID: \"542d6180-0507-48b6-bd4e-a79b055b1195\") " pod="opendatahub/maas-controller-6b5b68ff8f-9kmqm"
Oct 30 01:24:39.312889 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.312730    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/542d6180-0507-48b6-bd4e-a79b055b1195-webhook-cert\") pod \"maas-controller-6b5b68ff8f-9kmqm\" (UID: \"542d6180-0507-48b6-bd4e-a79b055b1195\") " pod="opendatahub/maas-controller-6b5b68ff8f-9kmqm"
Oct 30 01:24:39.335235 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.335197    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h9f2\" (UniqueName: \"kubernetes.io/projected/542d6180-0507-48b6-bd4e-a79b055b1195-kube-api-access-6h9f2\") pod \"maas-controller-6b5b68ff8f-9kmqm\" (UID: \"542d6180-0507-48b6-bd4e-a79b055b1195\") " pod="opendatahub/maas-controller-6b5b68ff8f-9kmqm"
Oct 30 01:24:39.422897 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.422861    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b5b68ff8f-9kmqm"
Oct 30 01:24:39.723956 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.723882    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7df9692e-c210-4300-8bcd-6fc93f7dd2fe" path="/var/lib/kubelet/pods/7df9692e-c210-4300-8bcd-6fc93f7dd2fe/volumes"
Oct 30 01:24:39.724356 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.724341    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f42086fc-bfe1-4ded-be68-a54e07853fd2" path="/var/lib/kubelet/pods/f42086fc-bfe1-4ded-be68-a54e07853fd2/volumes"
Oct 30 01:24:39.725952 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.725927    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b5b68ff8f-9kmqm"]
Oct 30 01:24:39.727849 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:24:39.727826    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod542d6180_0507_48b6_bd4e_a79b055b1195.slice/crio-7cefdffce166863af7e79633aeeb381c942259ee7f05fbc72e61fd79812e60b9 WatchSource:0}: Error finding container 7cefdffce166863af7e79633aeeb381c942259ee7f05fbc72e61fd79812e60b9: Status 404 returned error can't find the container with id 7cefdffce166863af7e79633aeeb381c942259ee7f05fbc72e61fd79812e60b9
Oct 30 01:24:39.729268 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:39.729250    4393 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 01:24:40.089139 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:40.089033    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b5b68ff8f-9kmqm" event={"ID":"542d6180-0507-48b6-bd4e-a79b055b1195","Type":"ContainerStarted","Data":"7cefdffce166863af7e79633aeeb381c942259ee7f05fbc72e61fd79812e60b9"}
Oct 30 01:24:41.097046 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:41.097006    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b5b68ff8f-9kmqm" event={"ID":"542d6180-0507-48b6-bd4e-a79b055b1195","Type":"ContainerStarted","Data":"5083c0ad85991ef6e6d623cacb50dd4b434ec80d76efc25b0a4a84fdb4941a20"}
Oct 30 01:24:41.097534 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:41.097068    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6b5b68ff8f-9kmqm"
Oct 30 01:24:41.138937 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:41.138878    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6b5b68ff8f-9kmqm" podStartSLOduration=1.7356115970000001 podStartE2EDuration="2.138863078s" totalImagesPullingTime="403.251481ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:24:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:24:41.137970187 +0000 UTC m=+3171.525646576" watchObservedRunningTime="2026-10-30 01:24:41.138863078 +0000 UTC m=+3171.526539467"
Oct 30 01:24:52.105310 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:24:52.105282    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6b5b68ff8f-9kmqm"
Oct 30 01:24:56.512666 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:24:56.512615    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:25:12.404001 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:12.403957    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b5b68ff8f-9kmqm"]
Oct 30 01:25:12.404611 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:12.404193    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6b5b68ff8f-9kmqm" podUID="542d6180-0507-48b6-bd4e-a79b055b1195" containerName="manager" containerID="cri-o://5083c0ad85991ef6e6d623cacb50dd4b434ec80d76efc25b0a4a84fdb4941a20" gracePeriod=30
Oct 30 01:25:12.661563 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:12.661492    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b5b68ff8f-9kmqm"
Oct 30 01:25:12.759875 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:12.759841    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/542d6180-0507-48b6-bd4e-a79b055b1195-kube-api-access-6h9f2\" (UniqueName: \"kubernetes.io/projected/542d6180-0507-48b6-bd4e-a79b055b1195-kube-api-access-6h9f2\") pod \"542d6180-0507-48b6-bd4e-a79b055b1195\" (UID: \"542d6180-0507-48b6-bd4e-a79b055b1195\") "
Oct 30 01:25:12.760105 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:12.759930    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/542d6180-0507-48b6-bd4e-a79b055b1195-metrics-certs\" (UniqueName: \"kubernetes.io/secret/542d6180-0507-48b6-bd4e-a79b055b1195-metrics-certs\") pod \"542d6180-0507-48b6-bd4e-a79b055b1195\" (UID: \"542d6180-0507-48b6-bd4e-a79b055b1195\") "
Oct 30 01:25:12.760105 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:12.759966    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/542d6180-0507-48b6-bd4e-a79b055b1195-webhook-cert\" (UniqueName: \"kubernetes.io/secret/542d6180-0507-48b6-bd4e-a79b055b1195-webhook-cert\") pod \"542d6180-0507-48b6-bd4e-a79b055b1195\" (UID: \"542d6180-0507-48b6-bd4e-a79b055b1195\") "
Oct 30 01:25:12.762271 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:12.762238    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/542d6180-0507-48b6-bd4e-a79b055b1195-webhook-cert" pod "542d6180-0507-48b6-bd4e-a79b055b1195" (UID: "542d6180-0507-48b6-bd4e-a79b055b1195"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:12.762427 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:12.762316    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/542d6180-0507-48b6-bd4e-a79b055b1195-kube-api-access-6h9f2" pod "542d6180-0507-48b6-bd4e-a79b055b1195" (UID: "542d6180-0507-48b6-bd4e-a79b055b1195"). InnerVolumeSpecName "kube-api-access-6h9f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:25:12.762427 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:12.762347    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/542d6180-0507-48b6-bd4e-a79b055b1195-metrics-certs" pod "542d6180-0507-48b6-bd4e-a79b055b1195" (UID: "542d6180-0507-48b6-bd4e-a79b055b1195"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:25:12.860939 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:12.860895    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h9f2\" (UniqueName: \"kubernetes.io/projected/542d6180-0507-48b6-bd4e-a79b055b1195-kube-api-access-6h9f2\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:25:12.860939 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:12.860932    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/542d6180-0507-48b6-bd4e-a79b055b1195-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:25:12.860939 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:12.860944    4393 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/542d6180-0507-48b6-bd4e-a79b055b1195-webhook-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:25:13.285380 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:13.285344    4393 generic.go:356] "Generic (PLEG): container finished" podID="542d6180-0507-48b6-bd4e-a79b055b1195" containerID="5083c0ad85991ef6e6d623cacb50dd4b434ec80d76efc25b0a4a84fdb4941a20" exitCode=0
Oct 30 01:25:13.285585 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:13.285404    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b5b68ff8f-9kmqm"
Oct 30 01:25:13.285585 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:13.285436    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b5b68ff8f-9kmqm" event={"ID":"542d6180-0507-48b6-bd4e-a79b055b1195","Type":"ContainerDied","Data":"5083c0ad85991ef6e6d623cacb50dd4b434ec80d76efc25b0a4a84fdb4941a20"}
Oct 30 01:25:13.285585 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:13.285502    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b5b68ff8f-9kmqm" event={"ID":"542d6180-0507-48b6-bd4e-a79b055b1195","Type":"ContainerDied","Data":"7cefdffce166863af7e79633aeeb381c942259ee7f05fbc72e61fd79812e60b9"}
Oct 30 01:25:13.285585 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:13.285532    4393 scope.go:122] "RemoveContainer" containerID="5083c0ad85991ef6e6d623cacb50dd4b434ec80d76efc25b0a4a84fdb4941a20"
Oct 30 01:25:13.295948 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:13.295925    4393 scope.go:122] "RemoveContainer" containerID="5083c0ad85991ef6e6d623cacb50dd4b434ec80d76efc25b0a4a84fdb4941a20"
Oct 30 01:25:13.296281 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:25:13.296255    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5083c0ad85991ef6e6d623cacb50dd4b434ec80d76efc25b0a4a84fdb4941a20\": container with ID starting with 5083c0ad85991ef6e6d623cacb50dd4b434ec80d76efc25b0a4a84fdb4941a20 not found: ID does not exist" containerID="5083c0ad85991ef6e6d623cacb50dd4b434ec80d76efc25b0a4a84fdb4941a20"
Oct 30 01:25:13.296330 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:13.296292    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5083c0ad85991ef6e6d623cacb50dd4b434ec80d76efc25b0a4a84fdb4941a20"} err="failed to get container status \"5083c0ad85991ef6e6d623cacb50dd4b434ec80d76efc25b0a4a84fdb4941a20\": rpc error: code = NotFound desc = could not find container \"5083c0ad85991ef6e6d623cacb50dd4b434ec80d76efc25b0a4a84fdb4941a20\": container with ID starting with 5083c0ad85991ef6e6d623cacb50dd4b434ec80d76efc25b0a4a84fdb4941a20 not found: ID does not exist"
Oct 30 01:25:13.334979 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:13.334943    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b5b68ff8f-9kmqm"]
Oct 30 01:25:13.346764 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:13.346733    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6b5b68ff8f-9kmqm"]
Oct 30 01:25:13.725159 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:13.725116    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="542d6180-0507-48b6-bd4e-a79b055b1195" path="/var/lib/kubelet/pods/542d6180-0507-48b6-bd4e-a79b055b1195/volumes"
Oct 30 01:25:14.612742 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:14.612703    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6b5b68ff8f-jrgsq"]
Oct 30 01:25:14.613268 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:14.613247    4393 cpu_manager.go:414] "containerMap: removing container" podUID="542d6180-0507-48b6-bd4e-a79b055b1195" containerName="manager"
Oct 30 01:25:14.613374 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:14.613270    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="542d6180-0507-48b6-bd4e-a79b055b1195" containerName="manager"
Oct 30 01:25:14.613436 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:14.613395    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="542d6180-0507-48b6-bd4e-a79b055b1195" containerName="manager"
Oct 30 01:25:14.618010 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:14.617981    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b5b68ff8f-jrgsq"
Oct 30 01:25:14.643905 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:14.643861    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 01:25:14.644110 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:14.643916    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 01:25:14.644110 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:14.643985    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vzmlw\""
Oct 30 01:25:14.644110 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:14.644061    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 01:25:14.646580 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:14.646550    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b5b68ff8f-jrgsq"]
Oct 30 01:25:14.681624 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:14.681586    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a06f7a71-cfbc-4a87-a668-bb3fa21ee2ef-metrics-certs\") pod \"maas-controller-6b5b68ff8f-jrgsq\" (UID: \"a06f7a71-cfbc-4a87-a668-bb3fa21ee2ef\") " pod="opendatahub/maas-controller-6b5b68ff8f-jrgsq"
Oct 30 01:25:14.681624 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:14.681636    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7gfw\" (UniqueName: \"kubernetes.io/projected/a06f7a71-cfbc-4a87-a668-bb3fa21ee2ef-kube-api-access-f7gfw\") pod \"maas-controller-6b5b68ff8f-jrgsq\" (UID: \"a06f7a71-cfbc-4a87-a668-bb3fa21ee2ef\") " pod="opendatahub/maas-controller-6b5b68ff8f-jrgsq"
Oct 30 01:25:14.681937 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:14.681818    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a06f7a71-cfbc-4a87-a668-bb3fa21ee2ef-webhook-cert\") pod \"maas-controller-6b5b68ff8f-jrgsq\" (UID: \"a06f7a71-cfbc-4a87-a668-bb3fa21ee2ef\") " pod="opendatahub/maas-controller-6b5b68ff8f-jrgsq"
Oct 30 01:25:14.783405 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:14.783356    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a06f7a71-cfbc-4a87-a668-bb3fa21ee2ef-metrics-certs\") pod \"maas-controller-6b5b68ff8f-jrgsq\" (UID: \"a06f7a71-cfbc-4a87-a668-bb3fa21ee2ef\") " pod="opendatahub/maas-controller-6b5b68ff8f-jrgsq"
Oct 30 01:25:14.783405 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:14.783405    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7gfw\" (UniqueName: \"kubernetes.io/projected/a06f7a71-cfbc-4a87-a668-bb3fa21ee2ef-kube-api-access-f7gfw\") pod \"maas-controller-6b5b68ff8f-jrgsq\" (UID: \"a06f7a71-cfbc-4a87-a668-bb3fa21ee2ef\") " pod="opendatahub/maas-controller-6b5b68ff8f-jrgsq"
Oct 30 01:25:14.784032 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:14.783469    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a06f7a71-cfbc-4a87-a668-bb3fa21ee2ef-webhook-cert\") pod \"maas-controller-6b5b68ff8f-jrgsq\" (UID: \"a06f7a71-cfbc-4a87-a668-bb3fa21ee2ef\") " pod="opendatahub/maas-controller-6b5b68ff8f-jrgsq"
Oct 30 01:25:14.786324 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:14.786291    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a06f7a71-cfbc-4a87-a668-bb3fa21ee2ef-webhook-cert\") pod \"maas-controller-6b5b68ff8f-jrgsq\" (UID: \"a06f7a71-cfbc-4a87-a668-bb3fa21ee2ef\") " pod="opendatahub/maas-controller-6b5b68ff8f-jrgsq"
Oct 30 01:25:14.786460 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:14.786403    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a06f7a71-cfbc-4a87-a668-bb3fa21ee2ef-metrics-certs\") pod \"maas-controller-6b5b68ff8f-jrgsq\" (UID: \"a06f7a71-cfbc-4a87-a668-bb3fa21ee2ef\") " pod="opendatahub/maas-controller-6b5b68ff8f-jrgsq"
Oct 30 01:25:14.806731 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:14.806695    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7gfw\" (UniqueName: \"kubernetes.io/projected/a06f7a71-cfbc-4a87-a668-bb3fa21ee2ef-kube-api-access-f7gfw\") pod \"maas-controller-6b5b68ff8f-jrgsq\" (UID: \"a06f7a71-cfbc-4a87-a668-bb3fa21ee2ef\") " pod="opendatahub/maas-controller-6b5b68ff8f-jrgsq"
Oct 30 01:25:14.933250 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:14.933211    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b5b68ff8f-jrgsq"
Oct 30 01:25:15.238124 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:15.238095    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b5b68ff8f-jrgsq"]
Oct 30 01:25:15.240132 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:25:15.240093    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda06f7a71_cfbc_4a87_a668_bb3fa21ee2ef.slice/crio-67480f9474354598e287a1d4232808db06f210f6f9b8e5978cbe7f2141f1920f WatchSource:0}: Error finding container 67480f9474354598e287a1d4232808db06f210f6f9b8e5978cbe7f2141f1920f: Status 404 returned error can't find the container with id 67480f9474354598e287a1d4232808db06f210f6f9b8e5978cbe7f2141f1920f
Oct 30 01:25:15.300280 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:15.300243    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b5b68ff8f-jrgsq" event={"ID":"a06f7a71-cfbc-4a87-a668-bb3fa21ee2ef","Type":"ContainerStarted","Data":"67480f9474354598e287a1d4232808db06f210f6f9b8e5978cbe7f2141f1920f"}
Oct 30 01:25:16.306993 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:16.306952    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b5b68ff8f-jrgsq" event={"ID":"a06f7a71-cfbc-4a87-a668-bb3fa21ee2ef","Type":"ContainerStarted","Data":"f033e310e3b73168582898b6b8a12d357cb38fc0d750757cb9b38be6013eec5b"}
Oct 30 01:25:16.307462 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:16.307023    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6b5b68ff8f-jrgsq"
Oct 30 01:25:16.349664 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:16.349607    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6b5b68ff8f-jrgsq" podStartSLOduration=1.997688853 podStartE2EDuration="2.349592089s" totalImagesPullingTime="351.903236ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:16.34801928 +0000 UTC m=+3206.735695671" watchObservedRunningTime="2026-10-30 01:25:16.349592089 +0000 UTC m=+3206.737268477"
Oct 30 01:25:19.991445 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:19.991404    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk"]
Oct 30 01:25:19.995386 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:19.995361    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk"
Oct 30 01:25:20.017444 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:20.017401    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk"]
Oct 30 01:25:20.138465 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:20.138414    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58e0c954-cdcc-4ad1-bbf7-a2197816e842-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-26zqk\" (UID: \"58e0c954-cdcc-4ad1-bbf7-a2197816e842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk"
Oct 30 01:25:20.138679 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:20.138482    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh9jc\" (UniqueName: \"kubernetes.io/projected/58e0c954-cdcc-4ad1-bbf7-a2197816e842-kube-api-access-mh9jc\") pod \"maas-default-gateway-openshift-default-7d9897b54c-26zqk\" (UID: \"58e0c954-cdcc-4ad1-bbf7-a2197816e842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk"
Oct 30 01:25:20.138679 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:20.138502    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e0c954-cdcc-4ad1-bbf7-a2197816e842-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-26zqk\" (UID: \"58e0c954-cdcc-4ad1-bbf7-a2197816e842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk"
Oct 30 01:25:20.138679 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:20.138631    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58e0c954-cdcc-4ad1-bbf7-a2197816e842-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-26zqk\" (UID: \"58e0c954-cdcc-4ad1-bbf7-a2197816e842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk"
Oct 30 01:25:20.138679 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:20.138676    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e0c954-cdcc-4ad1-bbf7-a2197816e842-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-26zqk\" (UID: \"58e0c954-cdcc-4ad1-bbf7-a2197816e842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk"
Oct 30 01:25:20.138904 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:20.138763    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58e0c954-cdcc-4ad1-bbf7-a2197816e842-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-26zqk\" (UID: \"58e0c954-cdcc-4ad1-bbf7-a2197816e842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk"
Oct 30 01:25:20.138904 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:20.138811    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58e0c954-cdcc-4ad1-bbf7-a2197816e842-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-26zqk\" (UID: \"58e0c954-cdcc-4ad1-bbf7-a2197816e842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk"
Oct 30 01:25:20.138904 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:20.138832    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58e0c954-cdcc-4ad1-bbf7-a2197816e842-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-26zqk\" (UID: \"58e0c954-cdcc-4ad1-bbf7-a2197816e842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk"
Oct 30 01:25:20.138904 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:20.138850    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58e0c954-cdcc-4ad1-bbf7-a2197816e842-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-26zqk\" (UID: \"58e0c954-cdcc-4ad1-bbf7-a2197816e842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk"
Oct 30 01:25:20.240189 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:20.240139    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58e0c954-cdcc-4ad1-bbf7-a2197816e842-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-26zqk\" (UID: \"58e0c954-cdcc-4ad1-bbf7-a2197816e842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk"
Oct 30 01:25:20.240422 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:20.240199    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e0c954-cdcc-4ad1-bbf7-a2197816e842-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-26zqk\" (UID: \"58e0c954-cdcc-4ad1-bbf7-a2197816e842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk"
Oct 30 01:25:20.240422 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:20.240252    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58e0c954-cdcc-4ad1-bbf7-a2197816e842-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-26zqk\" (UID: \"58e0c954-cdcc-4ad1-bbf7-a2197816e842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk"
Oct 30 01:25:20.240422 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:20.240276    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58e0c954-cdcc-4ad1-bbf7-a2197816e842-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-26zqk\" (UID: \"58e0c954-cdcc-4ad1-bbf7-a2197816e842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk"
Oct 30 01:25:20.240622 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:20.240431    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58e0c954-cdcc-4ad1-bbf7-a2197816e842-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-26zqk\" (UID: \"58e0c954-cdcc-4ad1-bbf7-a2197816e842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk"
Oct 30 01:25:20.240622 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:20.240479    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58e0c954-cdcc-4ad1-bbf7-a2197816e842-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-26zqk\" (UID: \"58e0c954-cdcc-4ad1-bbf7-a2197816e842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk"
Oct 30 01:25:20.240622 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:20.240558    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58e0c954-cdcc-4ad1-bbf7-a2197816e842-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-26zqk\" (UID: \"58e0c954-cdcc-4ad1-bbf7-a2197816e842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk"
Oct 30 01:25:20.240622 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:20.240607    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh9jc\" (UniqueName: \"kubernetes.io/projected/58e0c954-cdcc-4ad1-bbf7-a2197816e842-kube-api-access-mh9jc\") pod \"maas-default-gateway-openshift-default-7d9897b54c-26zqk\" (UID: \"58e0c954-cdcc-4ad1-bbf7-a2197816e842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk"
Oct 30 01:25:20.240896 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:20.240637    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e0c954-cdcc-4ad1-bbf7-a2197816e842-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-26zqk\" (UID: \"58e0c954-cdcc-4ad1-bbf7-a2197816e842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk"
Oct 30 01:25:20.240896 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:20.240660    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58e0c954-cdcc-4ad1-bbf7-a2197816e842-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-26zqk\" (UID: \"58e0c954-cdcc-4ad1-bbf7-a2197816e842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk"
Oct 30 01:25:20.240896 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:20.240694    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58e0c954-cdcc-4ad1-bbf7-a2197816e842-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-26zqk\" (UID: \"58e0c954-cdcc-4ad1-bbf7-a2197816e842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk"
Oct 30 01:25:20.240896 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:20.240750    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e0c954-cdcc-4ad1-bbf7-a2197816e842-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-26zqk\" (UID: \"58e0c954-cdcc-4ad1-bbf7-a2197816e842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk"
Oct 30 01:25:20.241118 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:20.241099    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e0c954-cdcc-4ad1-bbf7-a2197816e842-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-26zqk\" (UID: \"58e0c954-cdcc-4ad1-bbf7-a2197816e842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk"
Oct 30 01:25:20.241186 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:20.241154    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58e0c954-cdcc-4ad1-bbf7-a2197816e842-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-26zqk\" (UID: \"58e0c954-cdcc-4ad1-bbf7-a2197816e842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk"
Oct 30 01:25:20.243023 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:20.242964    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58e0c954-cdcc-4ad1-bbf7-a2197816e842-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-26zqk\" (UID: \"58e0c954-cdcc-4ad1-bbf7-a2197816e842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk"
Oct 30 01:25:20.243578 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:20.243557    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58e0c954-cdcc-4ad1-bbf7-a2197816e842-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-26zqk\" (UID: \"58e0c954-cdcc-4ad1-bbf7-a2197816e842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk"
Oct 30 01:25:20.263831 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:20.263769    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58e0c954-cdcc-4ad1-bbf7-a2197816e842-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-26zqk\" (UID: \"58e0c954-cdcc-4ad1-bbf7-a2197816e842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk"
Oct 30 01:25:20.264031 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:20.263928    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh9jc\" (UniqueName: \"kubernetes.io/projected/58e0c954-cdcc-4ad1-bbf7-a2197816e842-kube-api-access-mh9jc\") pod \"maas-default-gateway-openshift-default-7d9897b54c-26zqk\" (UID: \"58e0c954-cdcc-4ad1-bbf7-a2197816e842\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk"
Oct 30 01:25:20.310019 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:20.309978    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk"
Oct 30 01:25:20.630144 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:20.630110    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk"]
Oct 30 01:25:20.631765 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:25:20.631737    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e0c954_cdcc_4ad1_bbf7_a2197816e842.slice/crio-f64a2cf1baf2af4085f4afe4dd3a3cc7f0b7dca3f4d599df717b7e664d1903fa WatchSource:0}: Error finding container f64a2cf1baf2af4085f4afe4dd3a3cc7f0b7dca3f4d599df717b7e664d1903fa: Status 404 returned error can't find the container with id f64a2cf1baf2af4085f4afe4dd3a3cc7f0b7dca3f4d599df717b7e664d1903fa
Oct 30 01:25:20.633979 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:20.633947    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:25:20.634042 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:20.634015    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:25:20.634084 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:20.634046    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:25:21.339594 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:21.339555    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk" event={"ID":"58e0c954-cdcc-4ad1-bbf7-a2197816e842","Type":"ContainerStarted","Data":"c5e43df99840d50b082a39fd620653ad3bffd77f9e271afe48dc1f2739644d46"}
Oct 30 01:25:21.340150 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:21.339615    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk" event={"ID":"58e0c954-cdcc-4ad1-bbf7-a2197816e842","Type":"ContainerStarted","Data":"f64a2cf1baf2af4085f4afe4dd3a3cc7f0b7dca3f4d599df717b7e664d1903fa"}
Oct 30 01:25:21.395139 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:21.395083    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk" podStartSLOduration=2.395067198 podStartE2EDuration="2.395067198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:21.394167052 +0000 UTC m=+3211.781843441" watchObservedRunningTime="2026-10-30 01:25:21.395067198 +0000 UTC m=+3211.782743587"
Oct 30 01:25:22.311194 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:22.311162    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk"
Oct 30 01:25:22.326953 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:22.326915    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-26zqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 30 01:25:22.327147 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:22.326979    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk" podUID="58e0c954-cdcc-4ad1-bbf7-a2197816e842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 30 01:25:23.311271 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:23.311229    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-26zqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 30 01:25:23.311957 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:23.311917    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk" podUID="58e0c954-cdcc-4ad1-bbf7-a2197816e842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 30 01:25:24.322808 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:24.318302    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-26zqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 30 01:25:24.322808 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:24.318375    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk" podUID="58e0c954-cdcc-4ad1-bbf7-a2197816e842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 30 01:25:25.322807 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:25.319253    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-26zqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 30 01:25:25.322807 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:25.319327    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk" podUID="58e0c954-cdcc-4ad1-bbf7-a2197816e842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 30 01:25:26.323808 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:26.323305    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-26zqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 30 01:25:26.323808 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:26.323381    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk" podUID="58e0c954-cdcc-4ad1-bbf7-a2197816e842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 30 01:25:26.535134 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:25:26.535088    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:25:27.310529 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:27.310485    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-26zqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 30 01:25:27.310771 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:27.310577    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk" podUID="58e0c954-cdcc-4ad1-bbf7-a2197816e842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 30 01:25:27.315947 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:27.315918    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6b5b68ff8f-jrgsq"
Oct 30 01:25:28.979095 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:28.979057    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk"
Oct 30 01:25:28.979616 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:28.979592    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk"
Oct 30 01:25:28.980497 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:25:28.980477    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk"
Oct 30 01:25:56.511571 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:25:56.511525    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:26:05.138459 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:05.138420    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56"]
Oct 30 01:26:05.143562 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:05.143528    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56"
Oct 30 01:26:05.168220 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:05.168182    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56"]
Oct 30 01:26:05.191803 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:05.191737    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4da6381-6d65-4f27-9f7f-907dd8b5da23-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-97m56\" (UID: \"d4da6381-6d65-4f27-9f7f-907dd8b5da23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56"
Oct 30 01:26:05.191803 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:05.191801    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4da6381-6d65-4f27-9f7f-907dd8b5da23-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-97m56\" (UID: \"d4da6381-6d65-4f27-9f7f-907dd8b5da23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56"
Oct 30 01:26:05.192164 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:05.191863    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4da6381-6d65-4f27-9f7f-907dd8b5da23-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-97m56\" (UID: \"d4da6381-6d65-4f27-9f7f-907dd8b5da23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56"
Oct 30 01:26:05.192164 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:05.191995    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4da6381-6d65-4f27-9f7f-907dd8b5da23-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-97m56\" (UID: \"d4da6381-6d65-4f27-9f7f-907dd8b5da23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56"
Oct 30 01:26:05.192164 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:05.192051    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4da6381-6d65-4f27-9f7f-907dd8b5da23-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-97m56\" (UID: \"d4da6381-6d65-4f27-9f7f-907dd8b5da23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56"
Oct 30 01:26:05.192164 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:05.192108    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4da6381-6d65-4f27-9f7f-907dd8b5da23-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-97m56\" (UID: \"d4da6381-6d65-4f27-9f7f-907dd8b5da23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56"
Oct 30 01:26:05.192444 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:05.192173    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4da6381-6d65-4f27-9f7f-907dd8b5da23-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-97m56\" (UID: \"d4da6381-6d65-4f27-9f7f-907dd8b5da23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56"
Oct 30 01:26:05.192444 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:05.192228    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm24f\" (UniqueName: \"kubernetes.io/projected/d4da6381-6d65-4f27-9f7f-907dd8b5da23-kube-api-access-zm24f\") pod \"maas-default-gateway-openshift-default-7d9897b54c-97m56\" (UID: \"d4da6381-6d65-4f27-9f7f-907dd8b5da23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56"
Oct 30 01:26:05.192444 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:05.192251    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4da6381-6d65-4f27-9f7f-907dd8b5da23-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-97m56\" (UID: \"d4da6381-6d65-4f27-9f7f-907dd8b5da23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56"
Oct 30 01:26:05.293480 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:05.293440    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm24f\" (UniqueName: \"kubernetes.io/projected/d4da6381-6d65-4f27-9f7f-907dd8b5da23-kube-api-access-zm24f\") pod \"maas-default-gateway-openshift-default-7d9897b54c-97m56\" (UID: \"d4da6381-6d65-4f27-9f7f-907dd8b5da23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56"
Oct 30 01:26:05.293480 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:05.293479    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4da6381-6d65-4f27-9f7f-907dd8b5da23-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-97m56\" (UID: \"d4da6381-6d65-4f27-9f7f-907dd8b5da23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56"
Oct 30 01:26:05.293755 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:05.293522    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4da6381-6d65-4f27-9f7f-907dd8b5da23-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-97m56\" (UID: \"d4da6381-6d65-4f27-9f7f-907dd8b5da23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56"
Oct 30 01:26:05.293755 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:05.293543    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4da6381-6d65-4f27-9f7f-907dd8b5da23-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-97m56\" (UID: \"d4da6381-6d65-4f27-9f7f-907dd8b5da23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56"
Oct 30 01:26:05.293755 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:05.293561    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4da6381-6d65-4f27-9f7f-907dd8b5da23-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-97m56\" (UID: \"d4da6381-6d65-4f27-9f7f-907dd8b5da23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56"
Oct 30 01:26:05.293755 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:05.293599    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4da6381-6d65-4f27-9f7f-907dd8b5da23-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-97m56\" (UID: \"d4da6381-6d65-4f27-9f7f-907dd8b5da23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56"
Oct 30 01:26:05.293755 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:05.293629    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4da6381-6d65-4f27-9f7f-907dd8b5da23-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-97m56\" (UID: \"d4da6381-6d65-4f27-9f7f-907dd8b5da23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56"
Oct 30 01:26:05.293755 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:05.293663    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4da6381-6d65-4f27-9f7f-907dd8b5da23-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-97m56\" (UID: \"d4da6381-6d65-4f27-9f7f-907dd8b5da23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56"
Oct 30 01:26:05.293755 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:05.293738    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4da6381-6d65-4f27-9f7f-907dd8b5da23-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-97m56\" (UID: \"d4da6381-6d65-4f27-9f7f-907dd8b5da23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56"
Oct 30 01:26:05.294158 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:05.294048    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4da6381-6d65-4f27-9f7f-907dd8b5da23-credential-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-97m56\" (UID: \"d4da6381-6d65-4f27-9f7f-907dd8b5da23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56"
Oct 30 01:26:05.294158 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:05.294075    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4da6381-6d65-4f27-9f7f-907dd8b5da23-istio-data\") pod \"maas-default-gateway-openshift-default-7d9897b54c-97m56\" (UID: \"d4da6381-6d65-4f27-9f7f-907dd8b5da23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56"
Oct 30 01:26:05.294266 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:05.294220    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4da6381-6d65-4f27-9f7f-907dd8b5da23-workload-socket\") pod \"maas-default-gateway-openshift-default-7d9897b54c-97m56\" (UID: \"d4da6381-6d65-4f27-9f7f-907dd8b5da23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56"
Oct 30 01:26:05.294323 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:05.294266    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4da6381-6d65-4f27-9f7f-907dd8b5da23-workload-certs\") pod \"maas-default-gateway-openshift-default-7d9897b54c-97m56\" (UID: \"d4da6381-6d65-4f27-9f7f-907dd8b5da23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56"
Oct 30 01:26:05.294585 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:05.294519    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4da6381-6d65-4f27-9f7f-907dd8b5da23-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7d9897b54c-97m56\" (UID: \"d4da6381-6d65-4f27-9f7f-907dd8b5da23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56"
Oct 30 01:26:05.296196 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:05.296165    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4da6381-6d65-4f27-9f7f-907dd8b5da23-istio-envoy\") pod \"maas-default-gateway-openshift-default-7d9897b54c-97m56\" (UID: \"d4da6381-6d65-4f27-9f7f-907dd8b5da23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56"
Oct 30 01:26:05.296349 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:05.296333    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4da6381-6d65-4f27-9f7f-907dd8b5da23-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7d9897b54c-97m56\" (UID: \"d4da6381-6d65-4f27-9f7f-907dd8b5da23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56"
Oct 30 01:26:05.316712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:05.316668    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4da6381-6d65-4f27-9f7f-907dd8b5da23-istio-token\") pod \"maas-default-gateway-openshift-default-7d9897b54c-97m56\" (UID: \"d4da6381-6d65-4f27-9f7f-907dd8b5da23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56"
Oct 30 01:26:05.316712 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:05.316712    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm24f\" (UniqueName: \"kubernetes.io/projected/d4da6381-6d65-4f27-9f7f-907dd8b5da23-kube-api-access-zm24f\") pod \"maas-default-gateway-openshift-default-7d9897b54c-97m56\" (UID: \"d4da6381-6d65-4f27-9f7f-907dd8b5da23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56"
Oct 30 01:26:05.457853 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:05.457748    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56"
Oct 30 01:26:05.776091 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:05.775967    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56"]
Oct 30 01:26:05.779372 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:26:05.779341    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4da6381_6d65_4f27_9f7f_907dd8b5da23.slice/crio-00d6bed1d70c6f9fb834dcf7cda04d0bb36295ea4234f71e31cf835d2d3e6376 WatchSource:0}: Error finding container 00d6bed1d70c6f9fb834dcf7cda04d0bb36295ea4234f71e31cf835d2d3e6376: Status 404 returned error can't find the container with id 00d6bed1d70c6f9fb834dcf7cda04d0bb36295ea4234f71e31cf835d2d3e6376
Oct 30 01:26:05.781666 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:05.781632    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:26:05.781807 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:05.781702    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:26:05.781807 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:05.781733    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:26:06.601575 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:06.601535    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56" event={"ID":"d4da6381-6d65-4f27-9f7f-907dd8b5da23","Type":"ContainerStarted","Data":"2ebbd6c4bbc9999ae2d34de7917d54157624c75e57b38a092ee58238fd1f57e0"}
Oct 30 01:26:06.602181 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:06.601624    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56" event={"ID":"d4da6381-6d65-4f27-9f7f-907dd8b5da23","Type":"ContainerStarted","Data":"00d6bed1d70c6f9fb834dcf7cda04d0bb36295ea4234f71e31cf835d2d3e6376"}
Oct 30 01:26:06.677564 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:06.677500    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56" podStartSLOduration=1.6774846060000002 podStartE2EDuration="1.677484606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:06.664838741 +0000 UTC m=+3257.052515132" watchObservedRunningTime="2026-10-30 01:26:06.677484606 +0000 UTC m=+3257.065160994"
Oct 30 01:26:07.458498 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:07.458460    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56"
Oct 30 01:26:07.460357 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:07.460328    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-97m56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 30 01:26:07.460509 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:07.460396    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56" podUID="d4da6381-6d65-4f27-9f7f-907dd8b5da23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 30 01:26:08.459100 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:08.459053    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-97m56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 30 01:26:08.459563 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:08.459131    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56" podUID="d4da6381-6d65-4f27-9f7f-907dd8b5da23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 30 01:26:09.465295 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:09.465250    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-97m56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 30 01:26:09.465997 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:09.465961    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56" podUID="d4da6381-6d65-4f27-9f7f-907dd8b5da23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 30 01:26:10.459049 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:10.459005    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-97m56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 30 01:26:10.459260 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:10.459069    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56" podUID="d4da6381-6d65-4f27-9f7f-907dd8b5da23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 30 01:26:11.458164 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:11.458126    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-97m56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 30 01:26:11.458682 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:11.458187    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56" podUID="d4da6381-6d65-4f27-9f7f-907dd8b5da23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 30 01:26:12.458838 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:12.458745    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-97m56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 30 01:26:12.459275 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:12.458873    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56" podUID="d4da6381-6d65-4f27-9f7f-907dd8b5da23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 30 01:26:14.459406 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:14.459354    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-97m56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:26:14.460030 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:14.459453    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56" podUID="d4da6381-6d65-4f27-9f7f-907dd8b5da23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:26:15.501019 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:15.500976    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-97m56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:26:15.501447 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:15.501046    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56" podUID="d4da6381-6d65-4f27-9f7f-907dd8b5da23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:26:16.238400 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:16.238368    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56"
Oct 30 01:26:16.238812 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:16.238748    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56"
Oct 30 01:26:16.239565 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:26:16.239542    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56"
Oct 30 01:26:26.510091 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:26:26.510045    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:26:56.511507 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:26:56.511457    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:27:00.163444 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:00.163403    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4"]
Oct 30 01:27:00.164022 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:00.163673    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4" podUID="0479058f-6642-46e3-83a2-2d32949309a1" containerName="discovery" containerID="cri-o://c83927fdc2bb88419d155b81f8aab56fbf1d44e6d48b1074aa7850dce1011914" gracePeriod=30
Oct 30 01:27:00.430065 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:00.429990    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4"
Oct 30 01:27:00.526914 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:00.526856    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0479058f-6642-46e3-83a2-2d32949309a1-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0479058f-6642-46e3-83a2-2d32949309a1-istio-csr-dns-cert\") pod \"0479058f-6642-46e3-83a2-2d32949309a1\" (UID: \"0479058f-6642-46e3-83a2-2d32949309a1\") "
Oct 30 01:27:00.526914 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:00.526912    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0479058f-6642-46e3-83a2-2d32949309a1-istio-token\" (UniqueName: \"kubernetes.io/projected/0479058f-6642-46e3-83a2-2d32949309a1-istio-token\") pod \"0479058f-6642-46e3-83a2-2d32949309a1\" (UID: \"0479058f-6642-46e3-83a2-2d32949309a1\") "
Oct 30 01:27:00.527210 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:00.526940    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0479058f-6642-46e3-83a2-2d32949309a1-kube-api-access-fb6kp\" (UniqueName: \"kubernetes.io/projected/0479058f-6642-46e3-83a2-2d32949309a1-kube-api-access-fb6kp\") pod \"0479058f-6642-46e3-83a2-2d32949309a1\" (UID: \"0479058f-6642-46e3-83a2-2d32949309a1\") "
Oct 30 01:27:00.527210 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:00.526984    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0479058f-6642-46e3-83a2-2d32949309a1-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0479058f-6642-46e3-83a2-2d32949309a1-istio-csr-ca-configmap\") pod \"0479058f-6642-46e3-83a2-2d32949309a1\" (UID: \"0479058f-6642-46e3-83a2-2d32949309a1\") "
Oct 30 01:27:00.527210 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:00.527056    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0479058f-6642-46e3-83a2-2d32949309a1-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0479058f-6642-46e3-83a2-2d32949309a1-local-certs\") pod \"0479058f-6642-46e3-83a2-2d32949309a1\" (UID: \"0479058f-6642-46e3-83a2-2d32949309a1\") "
Oct 30 01:27:00.527210 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:00.527165    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0479058f-6642-46e3-83a2-2d32949309a1-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0479058f-6642-46e3-83a2-2d32949309a1-istio-kubeconfig\") pod \"0479058f-6642-46e3-83a2-2d32949309a1\" (UID: \"0479058f-6642-46e3-83a2-2d32949309a1\") "
Oct 30 01:27:00.527433 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:00.527245    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0479058f-6642-46e3-83a2-2d32949309a1-cacerts\" (UniqueName: \"kubernetes.io/secret/0479058f-6642-46e3-83a2-2d32949309a1-cacerts\") pod \"0479058f-6642-46e3-83a2-2d32949309a1\" (UID: \"0479058f-6642-46e3-83a2-2d32949309a1\") "
Oct 30 01:27:00.527530 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:00.527492    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0479058f-6642-46e3-83a2-2d32949309a1-istio-csr-ca-configmap" pod "0479058f-6642-46e3-83a2-2d32949309a1" (UID: "0479058f-6642-46e3-83a2-2d32949309a1"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:27:00.527665 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:00.527646    4393 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0479058f-6642-46e3-83a2-2d32949309a1-istio-csr-ca-configmap\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:27:00.529741 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:00.529681    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0479058f-6642-46e3-83a2-2d32949309a1-istio-csr-dns-cert" pod "0479058f-6642-46e3-83a2-2d32949309a1" (UID: "0479058f-6642-46e3-83a2-2d32949309a1"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:27:00.529741 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:00.529706    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0479058f-6642-46e3-83a2-2d32949309a1-istio-token" pod "0479058f-6642-46e3-83a2-2d32949309a1" (UID: "0479058f-6642-46e3-83a2-2d32949309a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:00.530070 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:00.530023    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0479058f-6642-46e3-83a2-2d32949309a1-local-certs" pod "0479058f-6642-46e3-83a2-2d32949309a1" (UID: "0479058f-6642-46e3-83a2-2d32949309a1"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:27:00.530187 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:00.530086    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0479058f-6642-46e3-83a2-2d32949309a1-istio-kubeconfig" pod "0479058f-6642-46e3-83a2-2d32949309a1" (UID: "0479058f-6642-46e3-83a2-2d32949309a1"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:27:00.530187 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:00.530120    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0479058f-6642-46e3-83a2-2d32949309a1-cacerts" pod "0479058f-6642-46e3-83a2-2d32949309a1" (UID: "0479058f-6642-46e3-83a2-2d32949309a1"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:27:00.530187 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:00.530126    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0479058f-6642-46e3-83a2-2d32949309a1-kube-api-access-fb6kp" pod "0479058f-6642-46e3-83a2-2d32949309a1" (UID: "0479058f-6642-46e3-83a2-2d32949309a1"). InnerVolumeSpecName "kube-api-access-fb6kp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:27:00.628663 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:00.628619    4393 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0479058f-6642-46e3-83a2-2d32949309a1-local-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:27:00.628663 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:00.628656    4393 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0479058f-6642-46e3-83a2-2d32949309a1-istio-kubeconfig\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:27:00.628663 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:00.628668    4393 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0479058f-6642-46e3-83a2-2d32949309a1-cacerts\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:27:00.628975 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:00.628677    4393 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0479058f-6642-46e3-83a2-2d32949309a1-istio-csr-dns-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:27:00.628975 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:00.628689    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0479058f-6642-46e3-83a2-2d32949309a1-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:27:00.628975 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:00.628701    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb6kp\" (UniqueName: \"kubernetes.io/projected/0479058f-6642-46e3-83a2-2d32949309a1-kube-api-access-fb6kp\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:27:00.981471 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:00.981429    4393 generic.go:356] "Generic (PLEG): container finished" podID="0479058f-6642-46e3-83a2-2d32949309a1" containerID="c83927fdc2bb88419d155b81f8aab56fbf1d44e6d48b1074aa7850dce1011914" exitCode=0
Oct 30 01:27:00.981677 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:00.981484    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4"
Oct 30 01:27:00.981677 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:00.981508    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4" event={"ID":"0479058f-6642-46e3-83a2-2d32949309a1","Type":"ContainerDied","Data":"c83927fdc2bb88419d155b81f8aab56fbf1d44e6d48b1074aa7850dce1011914"}
Oct 30 01:27:00.981677 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:00.981570    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4" event={"ID":"0479058f-6642-46e3-83a2-2d32949309a1","Type":"ContainerDied","Data":"529cc17b3a0a40c5345d0464cc8dd63a9dcdffcfe5e4331b0946f42b8e294292"}
Oct 30 01:27:00.981677 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:00.981590    4393 scope.go:122] "RemoveContainer" containerID="c83927fdc2bb88419d155b81f8aab56fbf1d44e6d48b1074aa7850dce1011914"
Oct 30 01:27:01.001588 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:01.001564    4393 scope.go:122] "RemoveContainer" containerID="c83927fdc2bb88419d155b81f8aab56fbf1d44e6d48b1074aa7850dce1011914"
Oct 30 01:27:01.001927 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:27:01.001903    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c83927fdc2bb88419d155b81f8aab56fbf1d44e6d48b1074aa7850dce1011914\": container with ID starting with c83927fdc2bb88419d155b81f8aab56fbf1d44e6d48b1074aa7850dce1011914 not found: ID does not exist" containerID="c83927fdc2bb88419d155b81f8aab56fbf1d44e6d48b1074aa7850dce1011914"
Oct 30 01:27:01.002034 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:01.001936    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c83927fdc2bb88419d155b81f8aab56fbf1d44e6d48b1074aa7850dce1011914"} err="failed to get container status \"c83927fdc2bb88419d155b81f8aab56fbf1d44e6d48b1074aa7850dce1011914\": rpc error: code = NotFound desc = could not find container \"c83927fdc2bb88419d155b81f8aab56fbf1d44e6d48b1074aa7850dce1011914\": container with ID starting with c83927fdc2bb88419d155b81f8aab56fbf1d44e6d48b1074aa7850dce1011914 not found: ID does not exist"
Oct 30 01:27:01.031549 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:01.031510    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4"]
Oct 30 01:27:01.043306 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:01.043259    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5nr4"]
Oct 30 01:27:01.723689 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:01.723651    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0479058f-6642-46e3-83a2-2d32949309a1" path="/var/lib/kubelet/pods/0479058f-6642-46e3-83a2-2d32949309a1/volumes"
Oct 30 01:27:07.309319 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:07.309262    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-cngvb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:27:07.310039 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:07.310004    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb" podUID="f3bd5875-a930-4985-b820-efe272d30763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:22.299992 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:22.299951    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-cngvb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:27:22.300434 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:27:22.300035    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb" podUID="f3bd5875-a930-4985-b820-efe272d30763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:27:26.511165 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:27:26.511119    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:27:56.511939 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:27:56.511900    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:28:07.308030 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:28:07.307923    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-cngvb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:28:07.308856 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:28:07.308817    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb" podUID="f3bd5875-a930-4985-b820-efe272d30763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:28:26.510652 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:28:26.510612    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:28:56.511381 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:28:56.511336    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:29:26.512394 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:29:26.512348    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:29:44.287977 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.287923    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"]
Oct 30 01:29:44.288757 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.288717    4393 cpu_manager.go:414] "containerMap: removing container" podUID="0479058f-6642-46e3-83a2-2d32949309a1" containerName="discovery"
Oct 30 01:29:44.288757 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.288754    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0479058f-6642-46e3-83a2-2d32949309a1" containerName="discovery"
Oct 30 01:29:44.289036 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.289010    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="0479058f-6642-46e3-83a2-2d32949309a1" containerName="discovery"
Oct 30 01:29:44.294083 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.294052    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:29:44.313125 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.313095    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-3ab7f1-openshift-default-dockercfg-6f4tw\""
Oct 30 01:29:44.326663 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.326619    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"]
Oct 30 01:29:44.427404 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.427354    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-istio-envoy\") pod \"e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn\" (UID: \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\") " pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:29:44.427610 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.427506    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-istio-token\") pod \"e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn\" (UID: \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\") " pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:29:44.427610 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.427560    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-credential-socket\") pod \"e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn\" (UID: \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\") " pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:29:44.427708 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.427614    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-istiod-ca-cert\") pod \"e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn\" (UID: \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\") " pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:29:44.427708 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.427644    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-workload-socket\") pod \"e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn\" (UID: \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\") " pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:29:44.427708 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.427690    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-istio-data\") pod \"e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn\" (UID: \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\") " pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:29:44.427910 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.427731    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-istio-podinfo\") pod \"e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn\" (UID: \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\") " pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:29:44.427910 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.427773    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjzmg\" (UniqueName: \"kubernetes.io/projected/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-kube-api-access-bjzmg\") pod \"e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn\" (UID: \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\") " pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:29:44.427910 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.427855    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-workload-certs\") pod \"e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn\" (UID: \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\") " pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:29:44.529305 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.529256    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-istio-data\") pod \"e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn\" (UID: \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\") " pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:29:44.529517 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.529315    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-istio-podinfo\") pod \"e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn\" (UID: \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\") " pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:29:44.529517 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.529353    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjzmg\" (UniqueName: \"kubernetes.io/projected/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-kube-api-access-bjzmg\") pod \"e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn\" (UID: \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\") " pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:29:44.529517 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.529387    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-workload-certs\") pod \"e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn\" (UID: \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\") " pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:29:44.529517 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.529495    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-istio-envoy\") pod \"e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn\" (UID: \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\") " pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:29:44.529765 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.529528    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-istio-token\") pod \"e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn\" (UID: \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\") " pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:29:44.529765 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.529565    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-credential-socket\") pod \"e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn\" (UID: \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\") " pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:29:44.529765 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.529615    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-istiod-ca-cert\") pod \"e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn\" (UID: \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\") " pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:29:44.529765 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.529643    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-workload-socket\") pod \"e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn\" (UID: \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\") " pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:29:44.529765 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.529748    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-istio-data\") pod \"e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn\" (UID: \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\") " pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:29:44.530058 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.529899    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-workload-certs\") pod \"e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn\" (UID: \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\") " pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:29:44.530134 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.530087    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-credential-socket\") pod \"e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn\" (UID: \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\") " pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:29:44.530134 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.530112    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-workload-socket\") pod \"e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn\" (UID: \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\") " pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:29:44.530567 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.530502    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-istiod-ca-cert\") pod \"e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn\" (UID: \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\") " pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:29:44.532314 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.532285    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-istio-envoy\") pod \"e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn\" (UID: \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\") " pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:29:44.532532 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.532500    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-istio-podinfo\") pod \"e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn\" (UID: \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\") " pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:29:44.552831 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.552720    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-istio-token\") pod \"e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn\" (UID: \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\") " pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:29:44.552979 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.552842    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjzmg\" (UniqueName: \"kubernetes.io/projected/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-kube-api-access-bjzmg\") pod \"e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn\" (UID: \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\") " pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:29:44.611244 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.611199    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:29:44.929548 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.929512    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"]
Oct 30 01:29:44.931297 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:29:44.931271    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb72680c6_37e2_4b58_89fc_1e2eb3ea72a8.slice/crio-55603f5e0380b949a290ec0554c20720f325dfb1c4db85b18cc2ee96f2b50cb7 WatchSource:0}: Error finding container 55603f5e0380b949a290ec0554c20720f325dfb1c4db85b18cc2ee96f2b50cb7: Status 404 returned error can't find the container with id 55603f5e0380b949a290ec0554c20720f325dfb1c4db85b18cc2ee96f2b50cb7
Oct 30 01:29:44.933489 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.933454    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:29:44.933604 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.933525    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:29:44.933604 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:44.933556    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:29:45.063912 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:45.063872    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn" event={"ID":"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8","Type":"ContainerStarted","Data":"597d7874dfc0d9edd108173c5da34050b8fe8588ecdeffd4bef25419df7c5fe8"}
Oct 30 01:29:45.064133 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:45.063932    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn" event={"ID":"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8","Type":"ContainerStarted","Data":"55603f5e0380b949a290ec0554c20720f325dfb1c4db85b18cc2ee96f2b50cb7"}
Oct 30 01:29:45.108156 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:45.108100    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn" podStartSLOduration=1.108087043 podStartE2EDuration="1.108087043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:45.107973693 +0000 UTC m=+3475.495650083" watchObservedRunningTime="2026-10-30 01:29:45.108087043 +0000 UTC m=+3475.495763433"
Oct 30 01:29:45.611520 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:45.611477    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:29:45.617087 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:45.617057    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:29:46.069825 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:46.069794    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:29:46.070907 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:46.070884    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:29:49.124920 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.124882    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs"]
Oct 30 01:29:49.129004 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.128975    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs"
Oct 30 01:29:49.147727 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.147692    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-3ab7f1-serving-cert\""
Oct 30 01:29:49.162136 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.162101    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs"]
Oct 30 01:29:49.278158 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.278124    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33a5b03c-0702-4952-a0bf-d6892a546766-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs\" (UID: \"33a5b03c-0702-4952-a0bf-d6892a546766\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs"
Oct 30 01:29:49.278366 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.278171    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33a5b03c-0702-4952-a0bf-d6892a546766-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs\" (UID: \"33a5b03c-0702-4952-a0bf-d6892a546766\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs"
Oct 30 01:29:49.278425 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.278356    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a5b03c-0702-4952-a0bf-d6892a546766-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs\" (UID: \"33a5b03c-0702-4952-a0bf-d6892a546766\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs"
Oct 30 01:29:49.278507 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.278483    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgnwk\" (UniqueName: \"kubernetes.io/projected/33a5b03c-0702-4952-a0bf-d6892a546766-kube-api-access-fgnwk\") pod \"maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs\" (UID: \"33a5b03c-0702-4952-a0bf-d6892a546766\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs"
Oct 30 01:29:49.378921 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.378819    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ab7f1-f54d8jfdkd"]
Oct 30 01:29:49.379302 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.379276    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33a5b03c-0702-4952-a0bf-d6892a546766-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs\" (UID: \"33a5b03c-0702-4952-a0bf-d6892a546766\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs"
Oct 30 01:29:49.379387 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.379377    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a5b03c-0702-4952-a0bf-d6892a546766-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs\" (UID: \"33a5b03c-0702-4952-a0bf-d6892a546766\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs"
Oct 30 01:29:49.379448 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.379419    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgnwk\" (UniqueName: \"kubernetes.io/projected/33a5b03c-0702-4952-a0bf-d6892a546766-kube-api-access-fgnwk\") pod \"maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs\" (UID: \"33a5b03c-0702-4952-a0bf-d6892a546766\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs"
Oct 30 01:29:49.379506 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.379493    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33a5b03c-0702-4952-a0bf-d6892a546766-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs\" (UID: \"33a5b03c-0702-4952-a0bf-d6892a546766\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs"
Oct 30 01:29:49.380205 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.380174    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a5b03c-0702-4952-a0bf-d6892a546766-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs\" (UID: \"33a5b03c-0702-4952-a0bf-d6892a546766\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs"
Oct 30 01:29:49.382193 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.382168    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33a5b03c-0702-4952-a0bf-d6892a546766-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs\" (UID: \"33a5b03c-0702-4952-a0bf-d6892a546766\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs"
Oct 30 01:29:49.382323 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.382265    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33a5b03c-0702-4952-a0bf-d6892a546766-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs\" (UID: \"33a5b03c-0702-4952-a0bf-d6892a546766\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs"
Oct 30 01:29:49.388152 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.388133    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ab7f1-f54d8jfdkd"
Oct 30 01:29:49.402707 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.402156    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-d7e2d007-dockercfg-26cdf\""
Oct 30 01:29:49.402707 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.402699    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-3ab7f1\""
Oct 30 01:29:49.405253 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.405223    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ab7f1-f54d8jfdkd"]
Oct 30 01:29:49.407314 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.407286    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgnwk\" (UniqueName: \"kubernetes.io/projected/33a5b03c-0702-4952-a0bf-d6892a546766-kube-api-access-fgnwk\") pod \"maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs\" (UID: \"33a5b03c-0702-4952-a0bf-d6892a546766\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs"
Oct 30 01:29:49.446056 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.446021    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs"
Oct 30 01:29:49.480690 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.480646    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84842582-4be7-4130-836a-0d3b73fe5cfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3ab7f1-f54d8jfdkd\" (UID: \"84842582-4be7-4130-836a-0d3b73fe5cfa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ab7f1-f54d8jfdkd"
Oct 30 01:29:49.480953 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.480727    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxbg9\" (UniqueName: \"kubernetes.io/projected/84842582-4be7-4130-836a-0d3b73fe5cfa-kube-api-access-cxbg9\") pod \"payload-pre-processing-e2e-worker-single-main-3ab7f1-f54d8jfdkd\" (UID: \"84842582-4be7-4130-836a-0d3b73fe5cfa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ab7f1-f54d8jfdkd"
Oct 30 01:29:49.487342 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.487308    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3ab7f1-5f47d96747ls65"]
Oct 30 01:29:49.491811 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.491752    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ab7f1-5f47d96747ls65"
Oct 30 01:29:49.512562 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.512528    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3ab7f1-5f47d96747ls65"]
Oct 30 01:29:49.582367 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.582316    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84842582-4be7-4130-836a-0d3b73fe5cfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3ab7f1-f54d8jfdkd\" (UID: \"84842582-4be7-4130-836a-0d3b73fe5cfa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ab7f1-f54d8jfdkd"
Oct 30 01:29:49.582582 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.582381    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxbg9\" (UniqueName: \"kubernetes.io/projected/84842582-4be7-4130-836a-0d3b73fe5cfa-kube-api-access-cxbg9\") pod \"payload-pre-processing-e2e-worker-single-main-3ab7f1-f54d8jfdkd\" (UID: \"84842582-4be7-4130-836a-0d3b73fe5cfa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ab7f1-f54d8jfdkd"
Oct 30 01:29:49.582582 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.582423    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/530a8b2b-e4b5-418e-b2d3-b7bf039ad428-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-3ab7f1-5f47d96747ls65\" (UID: \"530a8b2b-e4b5-418e-b2d3-b7bf039ad428\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ab7f1-5f47d96747ls65"
Oct 30 01:29:49.582582 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.582455    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qmr8\" (UniqueName: \"kubernetes.io/projected/530a8b2b-e4b5-418e-b2d3-b7bf039ad428-kube-api-access-7qmr8\") pod \"payload-processing-e2e-worker-single-main-3ab7f1-5f47d96747ls65\" (UID: \"530a8b2b-e4b5-418e-b2d3-b7bf039ad428\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ab7f1-5f47d96747ls65"
Oct 30 01:29:49.583137 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.583112    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84842582-4be7-4130-836a-0d3b73fe5cfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3ab7f1-f54d8jfdkd\" (UID: \"84842582-4be7-4130-836a-0d3b73fe5cfa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ab7f1-f54d8jfdkd"
Oct 30 01:29:49.605603 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.605570    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxbg9\" (UniqueName: \"kubernetes.io/projected/84842582-4be7-4130-836a-0d3b73fe5cfa-kube-api-access-cxbg9\") pod \"payload-pre-processing-e2e-worker-single-main-3ab7f1-f54d8jfdkd\" (UID: \"84842582-4be7-4130-836a-0d3b73fe5cfa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ab7f1-f54d8jfdkd"
Oct 30 01:29:49.683981 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.683875    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/530a8b2b-e4b5-418e-b2d3-b7bf039ad428-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-3ab7f1-5f47d96747ls65\" (UID: \"530a8b2b-e4b5-418e-b2d3-b7bf039ad428\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ab7f1-5f47d96747ls65"
Oct 30 01:29:49.683981 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.683922    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qmr8\" (UniqueName: \"kubernetes.io/projected/530a8b2b-e4b5-418e-b2d3-b7bf039ad428-kube-api-access-7qmr8\") pod \"payload-processing-e2e-worker-single-main-3ab7f1-5f47d96747ls65\" (UID: \"530a8b2b-e4b5-418e-b2d3-b7bf039ad428\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ab7f1-5f47d96747ls65"
Oct 30 01:29:49.684647 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.684613    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/530a8b2b-e4b5-418e-b2d3-b7bf039ad428-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-3ab7f1-5f47d96747ls65\" (UID: \"530a8b2b-e4b5-418e-b2d3-b7bf039ad428\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ab7f1-5f47d96747ls65"
Oct 30 01:29:49.706824 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.706756    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qmr8\" (UniqueName: \"kubernetes.io/projected/530a8b2b-e4b5-418e-b2d3-b7bf039ad428-kube-api-access-7qmr8\") pod \"payload-processing-e2e-worker-single-main-3ab7f1-5f47d96747ls65\" (UID: \"530a8b2b-e4b5-418e-b2d3-b7bf039ad428\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ab7f1-5f47d96747ls65"
Oct 30 01:29:49.724509 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.724476    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ab7f1-f54d8jfdkd"
Oct 30 01:29:49.794676 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.790028    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs"]
Oct 30 01:29:49.797358 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.797329    4393 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 01:29:49.818354 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:49.818321    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ab7f1-5f47d96747ls65"
Oct 30 01:29:50.030608 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:50.030518    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ab7f1-f54d8jfdkd"]
Oct 30 01:29:50.033156 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:29:50.033125    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84842582_4be7_4130_836a_0d3b73fe5cfa.slice/crio-c200163ac45aa31c7579d5d24e594f62fb52139a36f63625006fc9a02cd7fa82 WatchSource:0}: Error finding container c200163ac45aa31c7579d5d24e594f62fb52139a36f63625006fc9a02cd7fa82: Status 404 returned error can't find the container with id c200163ac45aa31c7579d5d24e594f62fb52139a36f63625006fc9a02cd7fa82
Oct 30 01:29:50.093822 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:50.093757    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ab7f1-f54d8jfdkd" event={"ID":"84842582-4be7-4130-836a-0d3b73fe5cfa","Type":"ContainerStarted","Data":"c200163ac45aa31c7579d5d24e594f62fb52139a36f63625006fc9a02cd7fa82"}
Oct 30 01:29:50.095166 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:50.095138    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs" event={"ID":"33a5b03c-0702-4952-a0bf-d6892a546766","Type":"ContainerStarted","Data":"503a927b207e6edc7902aa75a7f7e0e7501a9cfbcbaee7253e43fde58b8ff110"}
Oct 30 01:29:50.122123 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:50.122087    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3ab7f1-5f47d96747ls65"]
Oct 30 01:29:50.122941 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:29:50.122908    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod530a8b2b_e4b5_418e_b2d3_b7bf039ad428.slice/crio-0cfca38df144c65529dfa9af5634e420e3e37622bc5a973ea70d333cd95d6f88 WatchSource:0}: Error finding container 0cfca38df144c65529dfa9af5634e420e3e37622bc5a973ea70d333cd95d6f88: Status 404 returned error can't find the container with id 0cfca38df144c65529dfa9af5634e420e3e37622bc5a973ea70d333cd95d6f88
Oct 30 01:29:51.105278 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:51.105235    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ab7f1-5f47d96747ls65" event={"ID":"530a8b2b-e4b5-418e-b2d3-b7bf039ad428","Type":"ContainerStarted","Data":"387e844fc1d043400c06877e596497f9ee566d2f7a00bebd2dbb667107cb2c36"}
Oct 30 01:29:51.105772 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:51.105308    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ab7f1-5f47d96747ls65" event={"ID":"530a8b2b-e4b5-418e-b2d3-b7bf039ad428","Type":"ContainerStarted","Data":"0cfca38df144c65529dfa9af5634e420e3e37622bc5a973ea70d333cd95d6f88"}
Oct 30 01:29:51.105772 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:51.105335    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ab7f1-5f47d96747ls65"
Oct 30 01:29:51.107875 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:51.107848    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ab7f1-f54d8jfdkd" event={"ID":"84842582-4be7-4130-836a-0d3b73fe5cfa","Type":"ContainerStarted","Data":"5386239bdb41f3951d24838bcf3b43a560bb5863cca401de52a82209f3a7912d"}
Oct 30 01:29:51.108038 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:51.107894    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ab7f1-f54d8jfdkd"
Oct 30 01:29:51.109582 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:51.109561    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs" event={"ID":"33a5b03c-0702-4952-a0bf-d6892a546766","Type":"ContainerStarted","Data":"1da2e711c1828d78b0bdcc0fca465484fac64f4a2ac7a90bd4e44ab5ee765898"}
Oct 30 01:29:51.109693 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:51.109674    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs"
Oct 30 01:29:51.146389 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:51.146336    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ab7f1-5f47d96747ls65" podStartSLOduration=2.146319018 podStartE2EDuration="2.146319018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:51.143928857 +0000 UTC m=+3481.531605250" watchObservedRunningTime="2026-10-30 01:29:51.146319018 +0000 UTC m=+3481.533995406"
Oct 30 01:29:51.184392 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:51.184341    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ab7f1-f54d8jfdkd" podStartSLOduration=2.184325868 podStartE2EDuration="2.184325868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:51.18206879 +0000 UTC m=+3481.569745179" watchObservedRunningTime="2026-10-30 01:29:51.184325868 +0000 UTC m=+3481.572002257"
Oct 30 01:29:51.224614 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:51.224554    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs" podStartSLOduration=1.755652071 podStartE2EDuration="2.224537099s" totalImagesPullingTime="468.885028ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:51.222476421 +0000 UTC m=+3481.610152834" watchObservedRunningTime="2026-10-30 01:29:51.224537099 +0000 UTC m=+3481.612213487"
Oct 30 01:29:56.544861 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:29:56.544817    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:29:57.130980 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:29:57.130942    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs"
Oct 30 01:30:01.825423 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:01.825380    4393 patch_prober.go:29] interesting pod/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": read tcp 10.133.0.2:56142->10.133.0.201:15021: read: connection reset by peer" start-of-body=
Oct 30 01:30:01.825921 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:01.825464    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn" podUID="b72680c6-37e2-4b58-89fc-1e2eb3ea72a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": read tcp 10.133.0.2:56142->10.133.0.201:15021: read: connection reset by peer"
Oct 30 01:30:02.127696 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:02.127667    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ab7f1-f54d8jfdkd"
Oct 30 01:30:02.127927 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:02.127749    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ab7f1-5f47d96747ls65"
Oct 30 01:30:02.203965 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:02.203926    4393 generic.go:356] "Generic (PLEG): container finished" podID="b72680c6-37e2-4b58-89fc-1e2eb3ea72a8" containerID="597d7874dfc0d9edd108173c5da34050b8fe8588ecdeffd4bef25419df7c5fe8" exitCode=0
Oct 30 01:30:02.204161 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:02.204007    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn" event={"ID":"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8","Type":"ContainerDied","Data":"597d7874dfc0d9edd108173c5da34050b8fe8588ecdeffd4bef25419df7c5fe8"}
Oct 30 01:30:02.204502 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:02.204485    4393 scope.go:122] "RemoveContainer" containerID="597d7874dfc0d9edd108173c5da34050b8fe8588ecdeffd4bef25419df7c5fe8"
Oct 30 01:30:02.205920 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:02.205890    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:30:02.206014 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:02.205979    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:30:02.206058 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:02.206011    4393 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 01:30:03.233182 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:03.233138    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn" event={"ID":"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8","Type":"ContainerStarted","Data":"57b6691fa690a8b465f18b1f224101b644c0d9f481613c5a951ba2da620042fc"}
Oct 30 01:30:03.621875 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:03.616258    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:30:03.621875 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:03.616404    4393 patch_prober.go:29] interesting pod/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 01:30:03.621875 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:03.616455    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn" podUID="b72680c6-37e2-4b58-89fc-1e2eb3ea72a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 01:30:04.619307 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:04.619261    4393 patch_prober.go:29] interesting pod/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 01:30:04.620040 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:04.620001    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn" podUID="b72680c6-37e2-4b58-89fc-1e2eb3ea72a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 01:30:04.878466 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:04.878364    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw"]
Oct 30 01:30:04.881369 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:04.881331    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw"
Oct 30 01:30:04.925806 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:04.925231    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-3ab7f1\"/\"kube-root-ca.crt\""
Oct 30 01:30:04.925806 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:04.925378    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-3ab7f1\"/\"openshift-service-ca.crt\""
Oct 30 01:30:04.935370 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:04.935331    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw"]
Oct 30 01:30:04.952931 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:04.952885    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2br9g\" (UniqueName: \"kubernetes.io/projected/25d4f083-270c-4798-8bcb-7313310a7423-kube-api-access-2br9g\") pod \"simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw\" (UID: \"25d4f083-270c-4798-8bcb-7313310a7423\") " pod="e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw"
Oct 30 01:30:04.952931 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:04.952931    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25d4f083-270c-4798-8bcb-7313310a7423-dshm\") pod \"simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw\" (UID: \"25d4f083-270c-4798-8bcb-7313310a7423\") " pod="e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw"
Oct 30 01:30:04.953168 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:04.952954    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25d4f083-270c-4798-8bcb-7313310a7423-tmp-dir\") pod \"simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw\" (UID: \"25d4f083-270c-4798-8bcb-7313310a7423\") " pod="e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw"
Oct 30 01:30:04.953168 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:04.953077    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25d4f083-270c-4798-8bcb-7313310a7423-model-cache\") pod \"simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw\" (UID: \"25d4f083-270c-4798-8bcb-7313310a7423\") " pod="e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw"
Oct 30 01:30:04.953238 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:04.953173    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25d4f083-270c-4798-8bcb-7313310a7423-tls-certs\") pod \"simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw\" (UID: \"25d4f083-270c-4798-8bcb-7313310a7423\") " pod="e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw"
Oct 30 01:30:04.953238 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:04.953202    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25d4f083-270c-4798-8bcb-7313310a7423-home\") pod \"simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw\" (UID: \"25d4f083-270c-4798-8bcb-7313310a7423\") " pod="e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw"
Oct 30 01:30:04.975440 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:04.975407    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-3ab7f1\"/\"default-dockercfg-vnrw5\""
Oct 30 01:30:04.975637 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:04.975548    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-3ab7f1\"/\"simulator-single-main-3ab7f1-kserve-self-signed-certs\""
Oct 30 01:30:05.054825 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:05.054760    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25d4f083-270c-4798-8bcb-7313310a7423-home\") pod \"simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw\" (UID: \"25d4f083-270c-4798-8bcb-7313310a7423\") " pod="e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw"
Oct 30 01:30:05.055032 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:05.054940    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2br9g\" (UniqueName: \"kubernetes.io/projected/25d4f083-270c-4798-8bcb-7313310a7423-kube-api-access-2br9g\") pod \"simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw\" (UID: \"25d4f083-270c-4798-8bcb-7313310a7423\") " pod="e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw"
Oct 30 01:30:05.055032 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:05.054977    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25d4f083-270c-4798-8bcb-7313310a7423-dshm\") pod \"simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw\" (UID: \"25d4f083-270c-4798-8bcb-7313310a7423\") " pod="e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw"
Oct 30 01:30:05.055032 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:05.055004    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25d4f083-270c-4798-8bcb-7313310a7423-tmp-dir\") pod \"simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw\" (UID: \"25d4f083-270c-4798-8bcb-7313310a7423\") " pod="e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw"
Oct 30 01:30:05.055210 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:05.055156    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25d4f083-270c-4798-8bcb-7313310a7423-model-cache\") pod \"simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw\" (UID: \"25d4f083-270c-4798-8bcb-7313310a7423\") " pod="e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw"
Oct 30 01:30:05.055286 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:05.055163    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25d4f083-270c-4798-8bcb-7313310a7423-home\") pod \"simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw\" (UID: \"25d4f083-270c-4798-8bcb-7313310a7423\") " pod="e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw"
Oct 30 01:30:05.055349 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:05.055286    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25d4f083-270c-4798-8bcb-7313310a7423-tls-certs\") pod \"simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw\" (UID: \"25d4f083-270c-4798-8bcb-7313310a7423\") " pod="e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw"
Oct 30 01:30:05.055430 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:05.055410    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25d4f083-270c-4798-8bcb-7313310a7423-tmp-dir\") pod \"simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw\" (UID: \"25d4f083-270c-4798-8bcb-7313310a7423\") " pod="e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw"
Oct 30 01:30:05.055496 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:05.055415    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25d4f083-270c-4798-8bcb-7313310a7423-model-cache\") pod \"simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw\" (UID: \"25d4f083-270c-4798-8bcb-7313310a7423\") " pod="e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw"
Oct 30 01:30:05.057342 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:05.057323    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25d4f083-270c-4798-8bcb-7313310a7423-dshm\") pod \"simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw\" (UID: \"25d4f083-270c-4798-8bcb-7313310a7423\") " pod="e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw"
Oct 30 01:30:05.057856 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:05.057839    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25d4f083-270c-4798-8bcb-7313310a7423-tls-certs\") pod \"simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw\" (UID: \"25d4f083-270c-4798-8bcb-7313310a7423\") " pod="e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw"
Oct 30 01:30:05.077758 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:05.077728    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2br9g\" (UniqueName: \"kubernetes.io/projected/25d4f083-270c-4798-8bcb-7313310a7423-kube-api-access-2br9g\") pod \"simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw\" (UID: \"25d4f083-270c-4798-8bcb-7313310a7423\") " pod="e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw"
Oct 30 01:30:05.194336 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:05.194224    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw"
Oct 30 01:30:05.534084 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:05.533976    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw"]
Oct 30 01:30:05.542747 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:30:05.542705    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25d4f083_270c_4798_8bcb_7313310a7423.slice/crio-ac03f55ca927c12d619cdd9bc71ba11d44c9c69140b5bf9c952093f3a02393b9 WatchSource:0}: Error finding container ac03f55ca927c12d619cdd9bc71ba11d44c9c69140b5bf9c952093f3a02393b9: Status 404 returned error can't find the container with id ac03f55ca927c12d619cdd9bc71ba11d44c9c69140b5bf9c952093f3a02393b9
Oct 30 01:30:05.621806 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:05.620238    4393 patch_prober.go:29] interesting pod/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 01:30:05.621806 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:05.620323    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn" podUID="b72680c6-37e2-4b58-89fc-1e2eb3ea72a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 01:30:06.254133 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:06.254086    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw" event={"ID":"25d4f083-270c-4798-8bcb-7313310a7423","Type":"ContainerStarted","Data":"3f950597b104eb69dfca6e5004aa125bbfcd6386c10920be95777688952775ad"}
Oct 30 01:30:06.254133 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:06.254140    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw" event={"ID":"25d4f083-270c-4798-8bcb-7313310a7423","Type":"ContainerStarted","Data":"ac03f55ca927c12d619cdd9bc71ba11d44c9c69140b5bf9c952093f3a02393b9"}
Oct 30 01:30:06.296283 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:06.296223    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw" podStartSLOduration=2.296205457 podStartE2EDuration="2.296205457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:06.294007037 +0000 UTC m=+3496.681683427" watchObservedRunningTime="2026-10-30 01:30:06.296205457 +0000 UTC m=+3496.683881840"
Oct 30 01:30:06.620872 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:06.619259    4393 patch_prober.go:29] interesting pod/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 01:30:06.620872 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:06.619326    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn" podUID="b72680c6-37e2-4b58-89fc-1e2eb3ea72a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 01:30:07.195023 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:07.194975    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw"
Oct 30 01:30:07.621874 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:07.616252    4393 patch_prober.go:29] interesting pod/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 30 01:30:07.621874 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:07.616323    4393 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn" podUID="b72680c6-37e2-4b58-89fc-1e2eb3ea72a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 30 01:30:08.870424 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:08.870381    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:30:08.871015 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:08.870995    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:30:08.871910 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:08.871887    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:30:15.195408 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:15.195369    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw"
Oct 30 01:30:15.208246 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:15.208221    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw"
Oct 30 01:30:15.324297 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:15.324261    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw"
Oct 30 01:30:26.512653 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:30:26.512611    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:30:44.170656 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:44.170614    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw"]
Oct 30 01:30:49.763914 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:49.763879    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3ab7f1-5f47d96747ls65"]
Oct 30 01:30:49.792706 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:49.792675    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ab7f1-f54d8jfdkd"]
Oct 30 01:30:52.307874 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:52.307836    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs"]
Oct 30 01:30:52.308436 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:52.308161    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs" podUID="33a5b03c-0702-4952-a0bf-d6892a546766" containerName="maas-api" containerID="cri-o://1da2e711c1828d78b0bdcc0fca465484fac64f4a2ac7a90bd4e44ab5ee765898" gracePeriod=30
Oct 30 01:30:52.782600 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:52.782572    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs"
Oct 30 01:30:52.900935 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:52.900903    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33a5b03c-0702-4952-a0bf-d6892a546766-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a5b03c-0702-4952-a0bf-d6892a546766-openshift-service-ca\") pod \"33a5b03c-0702-4952-a0bf-d6892a546766\" (UID: \"33a5b03c-0702-4952-a0bf-d6892a546766\") "
Oct 30 01:30:52.901164 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:52.901009    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33a5b03c-0702-4952-a0bf-d6892a546766-kube-api-access-fgnwk\" (UniqueName: \"kubernetes.io/projected/33a5b03c-0702-4952-a0bf-d6892a546766-kube-api-access-fgnwk\") pod \"33a5b03c-0702-4952-a0bf-d6892a546766\" (UID: \"33a5b03c-0702-4952-a0bf-d6892a546766\") "
Oct 30 01:30:52.901164 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:52.901052    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33a5b03c-0702-4952-a0bf-d6892a546766-metrics-certs\" (UniqueName: \"kubernetes.io/secret/33a5b03c-0702-4952-a0bf-d6892a546766-metrics-certs\") pod \"33a5b03c-0702-4952-a0bf-d6892a546766\" (UID: \"33a5b03c-0702-4952-a0bf-d6892a546766\") "
Oct 30 01:30:52.901164 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:52.901103    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33a5b03c-0702-4952-a0bf-d6892a546766-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33a5b03c-0702-4952-a0bf-d6892a546766-maas-api-tls\") pod \"33a5b03c-0702-4952-a0bf-d6892a546766\" (UID: \"33a5b03c-0702-4952-a0bf-d6892a546766\") "
Oct 30 01:30:52.901351 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:52.901327    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33a5b03c-0702-4952-a0bf-d6892a546766-openshift-service-ca" pod "33a5b03c-0702-4952-a0bf-d6892a546766" (UID: "33a5b03c-0702-4952-a0bf-d6892a546766"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:30:52.901574 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:52.901547    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a5b03c-0702-4952-a0bf-d6892a546766-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:30:52.903455 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:52.903426    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33a5b03c-0702-4952-a0bf-d6892a546766-maas-api-tls" pod "33a5b03c-0702-4952-a0bf-d6892a546766" (UID: "33a5b03c-0702-4952-a0bf-d6892a546766"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:30:52.903561 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:52.903483    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33a5b03c-0702-4952-a0bf-d6892a546766-metrics-certs" pod "33a5b03c-0702-4952-a0bf-d6892a546766" (UID: "33a5b03c-0702-4952-a0bf-d6892a546766"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:30:52.903561 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:52.903518    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a5b03c-0702-4952-a0bf-d6892a546766-kube-api-access-fgnwk" pod "33a5b03c-0702-4952-a0bf-d6892a546766" (UID: "33a5b03c-0702-4952-a0bf-d6892a546766"). InnerVolumeSpecName "kube-api-access-fgnwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:53.002651 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:53.002547    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgnwk\" (UniqueName: \"kubernetes.io/projected/33a5b03c-0702-4952-a0bf-d6892a546766-kube-api-access-fgnwk\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:30:53.002651 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:53.002620    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33a5b03c-0702-4952-a0bf-d6892a546766-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:30:53.002651 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:53.002640    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33a5b03c-0702-4952-a0bf-d6892a546766-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:30:53.545008 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:53.544966    4393 generic.go:356] "Generic (PLEG): container finished" podID="33a5b03c-0702-4952-a0bf-d6892a546766" containerID="1da2e711c1828d78b0bdcc0fca465484fac64f4a2ac7a90bd4e44ab5ee765898" exitCode=0
Oct 30 01:30:53.545509 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:53.545021    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs"
Oct 30 01:30:53.545509 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:53.545058    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs" event={"ID":"33a5b03c-0702-4952-a0bf-d6892a546766","Type":"ContainerDied","Data":"1da2e711c1828d78b0bdcc0fca465484fac64f4a2ac7a90bd4e44ab5ee765898"}
Oct 30 01:30:53.545509 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:53.545127    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs" event={"ID":"33a5b03c-0702-4952-a0bf-d6892a546766","Type":"ContainerDied","Data":"503a927b207e6edc7902aa75a7f7e0e7501a9cfbcbaee7253e43fde58b8ff110"}
Oct 30 01:30:53.545509 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:53.545163    4393 scope.go:122] "RemoveContainer" containerID="1da2e711c1828d78b0bdcc0fca465484fac64f4a2ac7a90bd4e44ab5ee765898"
Oct 30 01:30:53.558305 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:53.558284    4393 scope.go:122] "RemoveContainer" containerID="1da2e711c1828d78b0bdcc0fca465484fac64f4a2ac7a90bd4e44ab5ee765898"
Oct 30 01:30:53.558571 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:30:53.558553    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1da2e711c1828d78b0bdcc0fca465484fac64f4a2ac7a90bd4e44ab5ee765898\": container with ID starting with 1da2e711c1828d78b0bdcc0fca465484fac64f4a2ac7a90bd4e44ab5ee765898 not found: ID does not exist" containerID="1da2e711c1828d78b0bdcc0fca465484fac64f4a2ac7a90bd4e44ab5ee765898"
Oct 30 01:30:53.558620 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:53.558581    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1da2e711c1828d78b0bdcc0fca465484fac64f4a2ac7a90bd4e44ab5ee765898"} err="failed to get container status \"1da2e711c1828d78b0bdcc0fca465484fac64f4a2ac7a90bd4e44ab5ee765898\": rpc error: code = NotFound desc = could not find container \"1da2e711c1828d78b0bdcc0fca465484fac64f4a2ac7a90bd4e44ab5ee765898\": container with ID starting with 1da2e711c1828d78b0bdcc0fca465484fac64f4a2ac7a90bd4e44ab5ee765898 not found: ID does not exist"
Oct 30 01:30:53.594044 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:53.594017    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs"]
Oct 30 01:30:53.606057 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:53.606028    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3ab7f1-d8885cf4c-2zhvs"]
Oct 30 01:30:53.724519 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:53.724480    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33a5b03c-0702-4952-a0bf-d6892a546766" path="/var/lib/kubelet/pods/33a5b03c-0702-4952-a0bf-d6892a546766/volumes"
Oct 30 01:30:54.764328 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:54.764281    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ab7f1-5f47d96747ls65" podUID="530a8b2b-e4b5-418e-b2d3-b7bf039ad428" containerName="payload-processing" containerID="cri-o://387e844fc1d043400c06877e596497f9ee566d2f7a00bebd2dbb667107cb2c36" gracePeriod=25
Oct 30 01:30:54.793751 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:54.793704    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ab7f1-f54d8jfdkd" podUID="84842582-4be7-4130-836a-0d3b73fe5cfa" containerName="payload-pre-processing" containerID="cri-o://5386239bdb41f3951d24838bcf3b43a560bb5863cca401de52a82209f3a7912d" gracePeriod=25
Oct 30 01:30:56.512490 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:30:56.512451    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:30:59.192453 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:59.192399    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw" podUID="25d4f083-270c-4798-8bcb-7313310a7423" containerName="main" containerID="cri-o://3f950597b104eb69dfca6e5004aa125bbfcd6386c10920be95777688952775ad" gracePeriod=45
Oct 30 01:30:59.313687 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:59.313642    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw" podUID="25d4f083-270c-4798-8bcb-7313310a7423" containerName="main" probeResult="failure" output="Get \"https://10.133.0.205:8000/ready\": dial tcp 10.133.0.205:8000: connect: connection refused"
Oct 30 01:30:59.451701 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:59.451628    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw"
Oct 30 01:30:59.473074 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:59.473037    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d4f083-270c-4798-8bcb-7313310a7423-dshm\" (UniqueName: \"kubernetes.io/empty-dir/25d4f083-270c-4798-8bcb-7313310a7423-dshm\") pod \"25d4f083-270c-4798-8bcb-7313310a7423\" (UID: \"25d4f083-270c-4798-8bcb-7313310a7423\") "
Oct 30 01:30:59.473074 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:59.473081    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25d4f083-270c-4798-8bcb-7313310a7423-tls-certs\" (UniqueName: \"kubernetes.io/secret/25d4f083-270c-4798-8bcb-7313310a7423-tls-certs\") pod \"25d4f083-270c-4798-8bcb-7313310a7423\" (UID: \"25d4f083-270c-4798-8bcb-7313310a7423\") "
Oct 30 01:30:59.473349 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:59.473142    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d4f083-270c-4798-8bcb-7313310a7423-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25d4f083-270c-4798-8bcb-7313310a7423-tmp-dir\") pod \"25d4f083-270c-4798-8bcb-7313310a7423\" (UID: \"25d4f083-270c-4798-8bcb-7313310a7423\") "
Oct 30 01:30:59.473349 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:59.473184    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25d4f083-270c-4798-8bcb-7313310a7423-kube-api-access-2br9g\" (UniqueName: \"kubernetes.io/projected/25d4f083-270c-4798-8bcb-7313310a7423-kube-api-access-2br9g\") pod \"25d4f083-270c-4798-8bcb-7313310a7423\" (UID: \"25d4f083-270c-4798-8bcb-7313310a7423\") "
Oct 30 01:30:59.473349 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:59.473204    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d4f083-270c-4798-8bcb-7313310a7423-home\" (UniqueName: \"kubernetes.io/empty-dir/25d4f083-270c-4798-8bcb-7313310a7423-home\") pod \"25d4f083-270c-4798-8bcb-7313310a7423\" (UID: \"25d4f083-270c-4798-8bcb-7313310a7423\") "
Oct 30 01:30:59.473349 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:59.473306    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d4f083-270c-4798-8bcb-7313310a7423-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25d4f083-270c-4798-8bcb-7313310a7423-model-cache\") pod \"25d4f083-270c-4798-8bcb-7313310a7423\" (UID: \"25d4f083-270c-4798-8bcb-7313310a7423\") "
Oct 30 01:30:59.473592 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:59.473433    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d4f083-270c-4798-8bcb-7313310a7423-tmp-dir" pod "25d4f083-270c-4798-8bcb-7313310a7423" (UID: "25d4f083-270c-4798-8bcb-7313310a7423"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:59.473592 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:59.473506    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d4f083-270c-4798-8bcb-7313310a7423-home" pod "25d4f083-270c-4798-8bcb-7313310a7423" (UID: "25d4f083-270c-4798-8bcb-7313310a7423"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:59.473702 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:59.473632    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d4f083-270c-4798-8bcb-7313310a7423-model-cache" pod "25d4f083-270c-4798-8bcb-7313310a7423" (UID: "25d4f083-270c-4798-8bcb-7313310a7423"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:59.473759 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:59.473716    4393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25d4f083-270c-4798-8bcb-7313310a7423-tmp-dir\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:30:59.473759 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:59.473734    4393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25d4f083-270c-4798-8bcb-7313310a7423-home\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:30:59.473759 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:59.473744    4393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25d4f083-270c-4798-8bcb-7313310a7423-model-cache\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:30:59.475699 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:59.475670    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d4f083-270c-4798-8bcb-7313310a7423-dshm" pod "25d4f083-270c-4798-8bcb-7313310a7423" (UID: "25d4f083-270c-4798-8bcb-7313310a7423"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:30:59.475699 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:59.475687    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25d4f083-270c-4798-8bcb-7313310a7423-tls-certs" pod "25d4f083-270c-4798-8bcb-7313310a7423" (UID: "25d4f083-270c-4798-8bcb-7313310a7423"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:30:59.475699 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:59.475689    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d4f083-270c-4798-8bcb-7313310a7423-kube-api-access-2br9g" pod "25d4f083-270c-4798-8bcb-7313310a7423" (UID: "25d4f083-270c-4798-8bcb-7313310a7423"). InnerVolumeSpecName "kube-api-access-2br9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:30:59.575359 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:59.575318    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2br9g\" (UniqueName: \"kubernetes.io/projected/25d4f083-270c-4798-8bcb-7313310a7423-kube-api-access-2br9g\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:30:59.575359 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:59.575355    4393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25d4f083-270c-4798-8bcb-7313310a7423-dshm\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:30:59.575590 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:59.575368    4393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25d4f083-270c-4798-8bcb-7313310a7423-tls-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:30:59.582179 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:59.582145    4393 generic.go:356] "Generic (PLEG): container finished" podID="25d4f083-270c-4798-8bcb-7313310a7423" containerID="3f950597b104eb69dfca6e5004aa125bbfcd6386c10920be95777688952775ad" exitCode=0
Oct 30 01:30:59.582382 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:59.582201    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw"
Oct 30 01:30:59.582382 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:59.582223    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw" event={"ID":"25d4f083-270c-4798-8bcb-7313310a7423","Type":"ContainerDied","Data":"3f950597b104eb69dfca6e5004aa125bbfcd6386c10920be95777688952775ad"}
Oct 30 01:30:59.582382 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:59.582271    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw" event={"ID":"25d4f083-270c-4798-8bcb-7313310a7423","Type":"ContainerDied","Data":"ac03f55ca927c12d619cdd9bc71ba11d44c9c69140b5bf9c952093f3a02393b9"}
Oct 30 01:30:59.582382 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:59.582288    4393 scope.go:122] "RemoveContainer" containerID="3f950597b104eb69dfca6e5004aa125bbfcd6386c10920be95777688952775ad"
Oct 30 01:30:59.592993 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:59.592952    4393 scope.go:122] "RemoveContainer" containerID="3f950597b104eb69dfca6e5004aa125bbfcd6386c10920be95777688952775ad"
Oct 30 01:30:59.593320 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:30:59.593298    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f950597b104eb69dfca6e5004aa125bbfcd6386c10920be95777688952775ad\": container with ID starting with 3f950597b104eb69dfca6e5004aa125bbfcd6386c10920be95777688952775ad not found: ID does not exist" containerID="3f950597b104eb69dfca6e5004aa125bbfcd6386c10920be95777688952775ad"
Oct 30 01:30:59.593399 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:59.593333    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f950597b104eb69dfca6e5004aa125bbfcd6386c10920be95777688952775ad"} err="failed to get container status \"3f950597b104eb69dfca6e5004aa125bbfcd6386c10920be95777688952775ad\": rpc error: code = NotFound desc = could not find container \"3f950597b104eb69dfca6e5004aa125bbfcd6386c10920be95777688952775ad\": container with ID starting with 3f950597b104eb69dfca6e5004aa125bbfcd6386c10920be95777688952775ad not found: ID does not exist"
Oct 30 01:30:59.634752 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:59.634714    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw"]
Oct 30 01:30:59.640403 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:59.640368    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-3ab7f1/simulator-single-main-3ab7f1-kserve-75697d5488-tzzlw"]
Oct 30 01:30:59.673513 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:59.673479    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"]
Oct 30 01:30:59.673747 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:59.673720    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn" podUID="b72680c6-37e2-4b58-89fc-1e2eb3ea72a8" containerName="istio-proxy" containerID="cri-o://57b6691fa690a8b465f18b1f224101b644c0d9f481613c5a951ba2da620042fc" gracePeriod=30
Oct 30 01:30:59.723838 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:30:59.723730    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25d4f083-270c-4798-8bcb-7313310a7423" path="/var/lib/kubelet/pods/25d4f083-270c-4798-8bcb-7313310a7423/volumes"
Oct 30 01:31:02.126321 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:02.126229    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ab7f1-5f47d96747ls65" podUID="530a8b2b-e4b5-418e-b2d3-b7bf039ad428" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:31:02.126321 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:02.126257    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ab7f1-f54d8jfdkd" podUID="84842582-4be7-4130-836a-0d3b73fe5cfa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:31:04.934539 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:04.934510    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:31:05.036488 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.036374    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-credential-socket\") pod \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\" (UID: \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\") "
Oct 30 01:31:05.036488 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.036422    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-istio-data\") pod \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\" (UID: \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\") "
Oct 30 01:31:05.036964 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.036519    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-istio-podinfo\") pod \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\" (UID: \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\") "
Oct 30 01:31:05.036964 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.036549    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-istio-token\" (UniqueName: \"kubernetes.io/projected/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-istio-token\") pod \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\" (UID: \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\") "
Oct 30 01:31:05.036964 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.036595    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-istio-envoy\") pod \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\" (UID: \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\") "
Oct 30 01:31:05.036964 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.036635    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-workload-socket\") pod \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\" (UID: \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\") "
Oct 30 01:31:05.036964 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.036664    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-istiod-ca-cert\") pod \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\" (UID: \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\") "
Oct 30 01:31:05.036964 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.036707    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-kube-api-access-bjzmg\" (UniqueName: \"kubernetes.io/projected/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-kube-api-access-bjzmg\") pod \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\" (UID: \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\") "
Oct 30 01:31:05.036964 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.036754    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-credential-socket" pod "b72680c6-37e2-4b58-89fc-1e2eb3ea72a8" (UID: "b72680c6-37e2-4b58-89fc-1e2eb3ea72a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:31:05.036964 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.036758    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-workload-certs\") pod \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\" (UID: \"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8\") "
Oct 30 01:31:05.037410 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.036774    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-istio-data" pod "b72680c6-37e2-4b58-89fc-1e2eb3ea72a8" (UID: "b72680c6-37e2-4b58-89fc-1e2eb3ea72a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:31:05.037410 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.036985    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-workload-certs" pod "b72680c6-37e2-4b58-89fc-1e2eb3ea72a8" (UID: "b72680c6-37e2-4b58-89fc-1e2eb3ea72a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:31:05.037410 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.037082    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-workload-socket" pod "b72680c6-37e2-4b58-89fc-1e2eb3ea72a8" (UID: "b72680c6-37e2-4b58-89fc-1e2eb3ea72a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:31:05.037410 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.037129    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-istiod-ca-cert" pod "b72680c6-37e2-4b58-89fc-1e2eb3ea72a8" (UID: "b72680c6-37e2-4b58-89fc-1e2eb3ea72a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:31:05.037410 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.037241    4393 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-credential-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:31:05.039374 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.039342    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-kube-api-access-bjzmg" pod "b72680c6-37e2-4b58-89fc-1e2eb3ea72a8" (UID: "b72680c6-37e2-4b58-89fc-1e2eb3ea72a8"). InnerVolumeSpecName "kube-api-access-bjzmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:05.039524 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.039377    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-istio-token" pod "b72680c6-37e2-4b58-89fc-1e2eb3ea72a8" (UID: "b72680c6-37e2-4b58-89fc-1e2eb3ea72a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:05.039763 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.039733    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-istio-podinfo" pod "b72680c6-37e2-4b58-89fc-1e2eb3ea72a8" (UID: "b72680c6-37e2-4b58-89fc-1e2eb3ea72a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 01:31:05.039763 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.039752    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-istio-envoy" pod "b72680c6-37e2-4b58-89fc-1e2eb3ea72a8" (UID: "b72680c6-37e2-4b58-89fc-1e2eb3ea72a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:31:05.138254 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.138211    4393 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-istio-envoy\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:31:05.138254 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.138246    4393 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-workload-socket\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:31:05.138254 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.138255    4393 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-istiod-ca-cert\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:31:05.138254 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.138264    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjzmg\" (UniqueName: \"kubernetes.io/projected/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-kube-api-access-bjzmg\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:31:05.138564 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.138274    4393 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-workload-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:31:05.138564 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.138281    4393 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-istio-data\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:31:05.138564 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.138289    4393 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-istio-podinfo\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:31:05.138564 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.138296    4393 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8-istio-token\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:31:05.621076 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.621042    4393 generic.go:356] "Generic (PLEG): container finished" podID="b72680c6-37e2-4b58-89fc-1e2eb3ea72a8" containerID="57b6691fa690a8b465f18b1f224101b644c0d9f481613c5a951ba2da620042fc" exitCode=0
Oct 30 01:31:05.621378 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.621096    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"
Oct 30 01:31:05.621378 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.621135    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn" event={"ID":"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8","Type":"ContainerDied","Data":"57b6691fa690a8b465f18b1f224101b644c0d9f481613c5a951ba2da620042fc"}
Oct 30 01:31:05.621378 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.621207    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn" event={"ID":"b72680c6-37e2-4b58-89fc-1e2eb3ea72a8","Type":"ContainerDied","Data":"55603f5e0380b949a290ec0554c20720f325dfb1c4db85b18cc2ee96f2b50cb7"}
Oct 30 01:31:05.621378 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.621279    4393 scope.go:122] "RemoveContainer" containerID="57b6691fa690a8b465f18b1f224101b644c0d9f481613c5a951ba2da620042fc"
Oct 30 01:31:05.631698 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.631675    4393 scope.go:122] "RemoveContainer" containerID="597d7874dfc0d9edd108173c5da34050b8fe8588ecdeffd4bef25419df7c5fe8"
Oct 30 01:31:05.640920 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.640895    4393 scope.go:122] "RemoveContainer" containerID="57b6691fa690a8b465f18b1f224101b644c0d9f481613c5a951ba2da620042fc"
Oct 30 01:31:05.641245 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:31:05.641218    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57b6691fa690a8b465f18b1f224101b644c0d9f481613c5a951ba2da620042fc\": container with ID starting with 57b6691fa690a8b465f18b1f224101b644c0d9f481613c5a951ba2da620042fc not found: ID does not exist" containerID="57b6691fa690a8b465f18b1f224101b644c0d9f481613c5a951ba2da620042fc"
Oct 30 01:31:05.641336 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.641254    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57b6691fa690a8b465f18b1f224101b644c0d9f481613c5a951ba2da620042fc"} err="failed to get container status \"57b6691fa690a8b465f18b1f224101b644c0d9f481613c5a951ba2da620042fc\": rpc error: code = NotFound desc = could not find container \"57b6691fa690a8b465f18b1f224101b644c0d9f481613c5a951ba2da620042fc\": container with ID starting with 57b6691fa690a8b465f18b1f224101b644c0d9f481613c5a951ba2da620042fc not found: ID does not exist"
Oct 30 01:31:05.641336 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.641273    4393 scope.go:122] "RemoveContainer" containerID="597d7874dfc0d9edd108173c5da34050b8fe8588ecdeffd4bef25419df7c5fe8"
Oct 30 01:31:05.641541 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:31:05.641525    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"597d7874dfc0d9edd108173c5da34050b8fe8588ecdeffd4bef25419df7c5fe8\": container with ID starting with 597d7874dfc0d9edd108173c5da34050b8fe8588ecdeffd4bef25419df7c5fe8 not found: ID does not exist" containerID="597d7874dfc0d9edd108173c5da34050b8fe8588ecdeffd4bef25419df7c5fe8"
Oct 30 01:31:05.641579 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.641546    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"597d7874dfc0d9edd108173c5da34050b8fe8588ecdeffd4bef25419df7c5fe8"} err="failed to get container status \"597d7874dfc0d9edd108173c5da34050b8fe8588ecdeffd4bef25419df7c5fe8\": rpc error: code = NotFound desc = could not find container \"597d7874dfc0d9edd108173c5da34050b8fe8588ecdeffd4bef25419df7c5fe8\": container with ID starting with 597d7874dfc0d9edd108173c5da34050b8fe8588ecdeffd4bef25419df7c5fe8 not found: ID does not exist"
Oct 30 01:31:05.683684 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.683645    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"]
Oct 30 01:31:05.687449 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.687418    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-3ab7f1-openshift-default-7884d75fd99dlrn"]
Oct 30 01:31:05.723896 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:05.723861    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b72680c6-37e2-4b58-89fc-1e2eb3ea72a8" path="/var/lib/kubelet/pods/b72680c6-37e2-4b58-89fc-1e2eb3ea72a8/volumes"
Oct 30 01:31:12.125919 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:12.125872    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ab7f1-f54d8jfdkd" podUID="84842582-4be7-4130-836a-0d3b73fe5cfa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:31:12.125919 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:12.125911    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ab7f1-5f47d96747ls65" podUID="530a8b2b-e4b5-418e-b2d3-b7bf039ad428" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:31:15.060864 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:15.033904    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-26zqk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:31:15.060864 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:15.033995    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk" podUID="58e0c954-cdcc-4ad1-bbf7-a2197816e842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:31:15.291543 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:15.291147    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ab7f1-5f47d96747ls65"
Oct 30 01:31:15.393529 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:15.393496    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ab7f1-f54d8jfdkd"
Oct 30 01:31:15.395603 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:15.395568    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/530a8b2b-e4b5-418e-b2d3-b7bf039ad428-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/530a8b2b-e4b5-418e-b2d3-b7bf039ad428-plugins-config-volume\") pod \"530a8b2b-e4b5-418e-b2d3-b7bf039ad428\" (UID: \"530a8b2b-e4b5-418e-b2d3-b7bf039ad428\") "
Oct 30 01:31:15.395888 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:15.395629    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/530a8b2b-e4b5-418e-b2d3-b7bf039ad428-kube-api-access-7qmr8\" (UniqueName: \"kubernetes.io/projected/530a8b2b-e4b5-418e-b2d3-b7bf039ad428-kube-api-access-7qmr8\") pod \"530a8b2b-e4b5-418e-b2d3-b7bf039ad428\" (UID: \"530a8b2b-e4b5-418e-b2d3-b7bf039ad428\") "
Oct 30 01:31:15.396415 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:15.396393    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/530a8b2b-e4b5-418e-b2d3-b7bf039ad428-plugins-config-volume" pod "530a8b2b-e4b5-418e-b2d3-b7bf039ad428" (UID: "530a8b2b-e4b5-418e-b2d3-b7bf039ad428"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:31:15.398812 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:15.398768    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/530a8b2b-e4b5-418e-b2d3-b7bf039ad428-kube-api-access-7qmr8" pod "530a8b2b-e4b5-418e-b2d3-b7bf039ad428" (UID: "530a8b2b-e4b5-418e-b2d3-b7bf039ad428"). InnerVolumeSpecName "kube-api-access-7qmr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:15.497449 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:15.497408    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84842582-4be7-4130-836a-0d3b73fe5cfa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84842582-4be7-4130-836a-0d3b73fe5cfa-plugins-config-volume\") pod \"84842582-4be7-4130-836a-0d3b73fe5cfa\" (UID: \"84842582-4be7-4130-836a-0d3b73fe5cfa\") "
Oct 30 01:31:15.497647 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:15.497566    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84842582-4be7-4130-836a-0d3b73fe5cfa-kube-api-access-cxbg9\" (UniqueName: \"kubernetes.io/projected/84842582-4be7-4130-836a-0d3b73fe5cfa-kube-api-access-cxbg9\") pod \"84842582-4be7-4130-836a-0d3b73fe5cfa\" (UID: \"84842582-4be7-4130-836a-0d3b73fe5cfa\") "
Oct 30 01:31:15.498145 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:15.498101    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84842582-4be7-4130-836a-0d3b73fe5cfa-plugins-config-volume" pod "84842582-4be7-4130-836a-0d3b73fe5cfa" (UID: "84842582-4be7-4130-836a-0d3b73fe5cfa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:31:15.498692 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:15.498666    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/530a8b2b-e4b5-418e-b2d3-b7bf039ad428-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:31:15.498826 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:15.498698    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qmr8\" (UniqueName: \"kubernetes.io/projected/530a8b2b-e4b5-418e-b2d3-b7bf039ad428-kube-api-access-7qmr8\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:31:15.498826 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:15.498713    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84842582-4be7-4130-836a-0d3b73fe5cfa-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:31:15.522214 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:15.522162    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84842582-4be7-4130-836a-0d3b73fe5cfa-kube-api-access-cxbg9" pod "84842582-4be7-4130-836a-0d3b73fe5cfa" (UID: "84842582-4be7-4130-836a-0d3b73fe5cfa"). InnerVolumeSpecName "kube-api-access-cxbg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:31:15.599332 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:15.599285    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxbg9\" (UniqueName: \"kubernetes.io/projected/84842582-4be7-4130-836a-0d3b73fe5cfa-kube-api-access-cxbg9\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:31:15.687914 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:15.687819    4393 generic.go:356] "Generic (PLEG): container finished" podID="530a8b2b-e4b5-418e-b2d3-b7bf039ad428" containerID="387e844fc1d043400c06877e596497f9ee566d2f7a00bebd2dbb667107cb2c36" exitCode=0
Oct 30 01:31:15.688110 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:15.687933    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ab7f1-5f47d96747ls65"
Oct 30 01:31:15.688110 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:15.687936    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ab7f1-5f47d96747ls65" event={"ID":"530a8b2b-e4b5-418e-b2d3-b7bf039ad428","Type":"ContainerDied","Data":"387e844fc1d043400c06877e596497f9ee566d2f7a00bebd2dbb667107cb2c36"}
Oct 30 01:31:15.688110 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:15.688022    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3ab7f1-5f47d96747ls65" event={"ID":"530a8b2b-e4b5-418e-b2d3-b7bf039ad428","Type":"ContainerDied","Data":"0cfca38df144c65529dfa9af5634e420e3e37622bc5a973ea70d333cd95d6f88"}
Oct 30 01:31:15.688110 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:15.688047    4393 scope.go:122] "RemoveContainer" containerID="387e844fc1d043400c06877e596497f9ee566d2f7a00bebd2dbb667107cb2c36"
Oct 30 01:31:15.689462 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:15.689432    4393 generic.go:356] "Generic (PLEG): container finished" podID="84842582-4be7-4130-836a-0d3b73fe5cfa" containerID="5386239bdb41f3951d24838bcf3b43a560bb5863cca401de52a82209f3a7912d" exitCode=0
Oct 30 01:31:15.689605 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:15.689494    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ab7f1-f54d8jfdkd"
Oct 30 01:31:15.689605 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:15.689496    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ab7f1-f54d8jfdkd" event={"ID":"84842582-4be7-4130-836a-0d3b73fe5cfa","Type":"ContainerDied","Data":"5386239bdb41f3951d24838bcf3b43a560bb5863cca401de52a82209f3a7912d"}
Oct 30 01:31:15.689751 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:15.689609    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ab7f1-f54d8jfdkd" event={"ID":"84842582-4be7-4130-836a-0d3b73fe5cfa","Type":"ContainerDied","Data":"c200163ac45aa31c7579d5d24e594f62fb52139a36f63625006fc9a02cd7fa82"}
Oct 30 01:31:15.702680 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:15.702657    4393 scope.go:122] "RemoveContainer" containerID="387e844fc1d043400c06877e596497f9ee566d2f7a00bebd2dbb667107cb2c36"
Oct 30 01:31:15.703298 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:31:15.703266    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"387e844fc1d043400c06877e596497f9ee566d2f7a00bebd2dbb667107cb2c36\": container with ID starting with 387e844fc1d043400c06877e596497f9ee566d2f7a00bebd2dbb667107cb2c36 not found: ID does not exist" containerID="387e844fc1d043400c06877e596497f9ee566d2f7a00bebd2dbb667107cb2c36"
Oct 30 01:31:15.703480 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:15.703310    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"387e844fc1d043400c06877e596497f9ee566d2f7a00bebd2dbb667107cb2c36"} err="failed to get container status \"387e844fc1d043400c06877e596497f9ee566d2f7a00bebd2dbb667107cb2c36\": rpc error: code = NotFound desc = could not find container \"387e844fc1d043400c06877e596497f9ee566d2f7a00bebd2dbb667107cb2c36\": container with ID starting with 387e844fc1d043400c06877e596497f9ee566d2f7a00bebd2dbb667107cb2c36 not found: ID does not exist"
Oct 30 01:31:15.703480 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:15.703328    4393 scope.go:122] "RemoveContainer" containerID="5386239bdb41f3951d24838bcf3b43a560bb5863cca401de52a82209f3a7912d"
Oct 30 01:31:15.712250 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:15.712228    4393 scope.go:122] "RemoveContainer" containerID="5386239bdb41f3951d24838bcf3b43a560bb5863cca401de52a82209f3a7912d"
Oct 30 01:31:15.712543 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:31:15.712523    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5386239bdb41f3951d24838bcf3b43a560bb5863cca401de52a82209f3a7912d\": container with ID starting with 5386239bdb41f3951d24838bcf3b43a560bb5863cca401de52a82209f3a7912d not found: ID does not exist" containerID="5386239bdb41f3951d24838bcf3b43a560bb5863cca401de52a82209f3a7912d"
Oct 30 01:31:15.712623 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:15.712552    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5386239bdb41f3951d24838bcf3b43a560bb5863cca401de52a82209f3a7912d"} err="failed to get container status \"5386239bdb41f3951d24838bcf3b43a560bb5863cca401de52a82209f3a7912d\": rpc error: code = NotFound desc = could not find container \"5386239bdb41f3951d24838bcf3b43a560bb5863cca401de52a82209f3a7912d\": container with ID starting with 5386239bdb41f3951d24838bcf3b43a560bb5863cca401de52a82209f3a7912d not found: ID does not exist"
Oct 30 01:31:15.751981 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:15.751941    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3ab7f1-5f47d96747ls65"]
Oct 30 01:31:15.756714 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:15.756682    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3ab7f1-5f47d96747ls65"]
Oct 30 01:31:15.791047 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:15.791015    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ab7f1-f54d8jfdkd"]
Oct 30 01:31:15.794501 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:15.794467    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3ab7f1-f54d8jfdkd"]
Oct 30 01:31:17.723578 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:17.723544    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="530a8b2b-e4b5-418e-b2d3-b7bf039ad428" path="/var/lib/kubelet/pods/530a8b2b-e4b5-418e-b2d3-b7bf039ad428/volumes"
Oct 30 01:31:17.724033 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:31:17.724013    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84842582-4be7-4130-836a-0d3b73fe5cfa" path="/var/lib/kubelet/pods/84842582-4be7-4130-836a-0d3b73fe5cfa/volumes"
Oct 30 01:31:26.511685 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:31:26.511644    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:31:56.511231 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:31:56.511189    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:32:26.511163 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:32:26.511122    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:32:31.702610 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:32:31.702510    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4zr2r"]
Oct 30 01:32:31.703086 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:32:31.702998    4393 cpu_manager.go:414] "containerMap: removing container" podUID="530a8b2b-e4b5-418e-b2d3-b7bf039ad428" containerName="payload-processing"
Oct 30 01:32:31.703086 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:32:31.703012    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="530a8b2b-e4b5-418e-b2d3-b7bf039ad428" containerName="payload-processing"
Oct 30 01:32:31.703086 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:32:31.703027    4393 cpu_manager.go:414] "containerMap: removing container" podUID="84842582-4be7-4130-836a-0d3b73fe5cfa" containerName="payload-pre-processing"
Oct 30 01:32:31.703086 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:32:31.703033    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84842582-4be7-4130-836a-0d3b73fe5cfa" containerName="payload-pre-processing"
Oct 30 01:32:31.703086 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:32:31.703052    4393 cpu_manager.go:414] "containerMap: removing container" podUID="b72680c6-37e2-4b58-89fc-1e2eb3ea72a8" containerName="istio-proxy"
Oct 30 01:32:31.703086 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:32:31.703057    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b72680c6-37e2-4b58-89fc-1e2eb3ea72a8" containerName="istio-proxy"
Oct 30 01:32:31.703086 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:32:31.703066    4393 cpu_manager.go:414] "containerMap: removing container" podUID="b72680c6-37e2-4b58-89fc-1e2eb3ea72a8" containerName="istio-proxy"
Oct 30 01:32:31.703086 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:32:31.703071    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b72680c6-37e2-4b58-89fc-1e2eb3ea72a8" containerName="istio-proxy"
Oct 30 01:32:31.703086 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:32:31.703091    4393 cpu_manager.go:414] "containerMap: removing container" podUID="33a5b03c-0702-4952-a0bf-d6892a546766" containerName="maas-api"
Oct 30 01:32:31.703381 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:32:31.703096    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33a5b03c-0702-4952-a0bf-d6892a546766" containerName="maas-api"
Oct 30 01:32:31.703381 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:32:31.703109    4393 cpu_manager.go:414] "containerMap: removing container" podUID="25d4f083-270c-4798-8bcb-7313310a7423" containerName="main"
Oct 30 01:32:31.703381 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:32:31.703114    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25d4f083-270c-4798-8bcb-7313310a7423" containerName="main"
Oct 30 01:32:31.703381 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:32:31.703174    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="b72680c6-37e2-4b58-89fc-1e2eb3ea72a8" containerName="istio-proxy"
Oct 30 01:32:31.703381 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:32:31.703184    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="b72680c6-37e2-4b58-89fc-1e2eb3ea72a8" containerName="istio-proxy"
Oct 30 01:32:31.703381 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:32:31.703192    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="33a5b03c-0702-4952-a0bf-d6892a546766" containerName="maas-api"
Oct 30 01:32:31.703381 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:32:31.703199    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="25d4f083-270c-4798-8bcb-7313310a7423" containerName="main"
Oct 30 01:32:31.703381 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:32:31.703204    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="530a8b2b-e4b5-418e-b2d3-b7bf039ad428" containerName="payload-processing"
Oct 30 01:32:31.703381 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:32:31.703210    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="84842582-4be7-4130-836a-0d3b73fe5cfa" containerName="payload-pre-processing"
Oct 30 01:32:31.707825 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:32:31.707713    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4zr2r"
Oct 30 01:32:31.728635 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:32:31.728603    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4zr2r"]
Oct 30 01:32:31.731021 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:32:31.730990    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-pj9zs\""
Oct 30 01:32:31.872164 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:32:31.872113    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c2d79a11-d151-4368-9469-2034d93cd39d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-4zr2r\" (UID: \"c2d79a11-d151-4368-9469-2034d93cd39d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4zr2r"
Oct 30 01:32:31.872164 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:32:31.872162    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmvkn\" (UniqueName: \"kubernetes.io/projected/c2d79a11-d151-4368-9469-2034d93cd39d-kube-api-access-mmvkn\") pod \"kuadrant-operator-controller-manager-57bb9bfff-4zr2r\" (UID: \"c2d79a11-d151-4368-9469-2034d93cd39d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4zr2r"
Oct 30 01:32:31.972848 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:32:31.972721    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c2d79a11-d151-4368-9469-2034d93cd39d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-4zr2r\" (UID: \"c2d79a11-d151-4368-9469-2034d93cd39d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4zr2r"
Oct 30 01:32:31.972848 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:32:31.972774    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmvkn\" (UniqueName: \"kubernetes.io/projected/c2d79a11-d151-4368-9469-2034d93cd39d-kube-api-access-mmvkn\") pod \"kuadrant-operator-controller-manager-57bb9bfff-4zr2r\" (UID: \"c2d79a11-d151-4368-9469-2034d93cd39d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4zr2r"
Oct 30 01:32:31.973136 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:32:31.973101    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c2d79a11-d151-4368-9469-2034d93cd39d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-4zr2r\" (UID: \"c2d79a11-d151-4368-9469-2034d93cd39d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4zr2r"
Oct 30 01:32:31.996281 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:32:31.996249    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmvkn\" (UniqueName: \"kubernetes.io/projected/c2d79a11-d151-4368-9469-2034d93cd39d-kube-api-access-mmvkn\") pod \"kuadrant-operator-controller-manager-57bb9bfff-4zr2r\" (UID: \"c2d79a11-d151-4368-9469-2034d93cd39d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4zr2r"
Oct 30 01:32:32.029007 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:32:32.028967    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4zr2r"
Oct 30 01:32:32.367360 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:32:32.367270    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4zr2r"]
Oct 30 01:32:32.380734 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:32:32.380699    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d79a11_d151_4368_9469_2034d93cd39d.slice/crio-f6e6428944aa9591f3198776c02b0fb720f5bebc4b48c196ee87c51be67e9517 WatchSource:0}: Error finding container f6e6428944aa9591f3198776c02b0fb720f5bebc4b48c196ee87c51be67e9517: Status 404 returned error can't find the container with id f6e6428944aa9591f3198776c02b0fb720f5bebc4b48c196ee87c51be67e9517
Oct 30 01:32:33.120986 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:32:33.120944    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4zr2r" event={"ID":"c2d79a11-d151-4368-9469-2034d93cd39d","Type":"ContainerStarted","Data":"9a96106d0f8797876b222eca826b07ec49428b88510312e3ebd463a9dec0b15f"}
Oct 30 01:32:33.121401 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:32:33.121011    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4zr2r" event={"ID":"c2d79a11-d151-4368-9469-2034d93cd39d","Type":"ContainerStarted","Data":"f6e6428944aa9591f3198776c02b0fb720f5bebc4b48c196ee87c51be67e9517"}
Oct 30 01:32:33.121401 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:32:33.121138    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4zr2r"
Oct 30 01:32:33.196995 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:32:33.196938    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4zr2r" podStartSLOduration=2.196920978 podStartE2EDuration="2.196920978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:33.18406231 +0000 UTC m=+3643.571738699" watchObservedRunningTime="2026-10-30 01:32:33.196920978 +0000 UTC m=+3643.584597369"
Oct 30 01:32:44.129246 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:32:44.129212    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-4zr2r"
Oct 30 01:32:56.511828 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:32:56.511761    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:33:26.513652 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:33:26.513608    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:33:29.952887 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:29.952845    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97"]
Oct 30 01:33:29.956814 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:29.956762    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97"
Oct 30 01:33:30.012182 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.012141    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-fe8356-serving-cert\""
Oct 30 01:33:30.013725 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.013695    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97"]
Oct 30 01:33:30.041944 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.041910    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf4a50c-471b-4019-bff6-404c1ce9fbce-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97\" (UID: \"3bf4a50c-471b-4019-bff6-404c1ce9fbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97"
Oct 30 01:33:30.041944 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.041952    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bf4a50c-471b-4019-bff6-404c1ce9fbce-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97\" (UID: \"3bf4a50c-471b-4019-bff6-404c1ce9fbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97"
Oct 30 01:33:30.042185 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.042085    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bf4a50c-471b-4019-bff6-404c1ce9fbce-metrics-certs\") pod \"maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97\" (UID: \"3bf4a50c-471b-4019-bff6-404c1ce9fbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97"
Oct 30 01:33:30.042224 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.042183    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7thx6\" (UniqueName: \"kubernetes.io/projected/3bf4a50c-471b-4019-bff6-404c1ce9fbce-kube-api-access-7thx6\") pod \"maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97\" (UID: \"3bf4a50c-471b-4019-bff6-404c1ce9fbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97"
Oct 30 01:33:30.143721 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.143682    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf4a50c-471b-4019-bff6-404c1ce9fbce-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97\" (UID: \"3bf4a50c-471b-4019-bff6-404c1ce9fbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97"
Oct 30 01:33:30.143721 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.143726    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bf4a50c-471b-4019-bff6-404c1ce9fbce-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97\" (UID: \"3bf4a50c-471b-4019-bff6-404c1ce9fbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97"
Oct 30 01:33:30.144042 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.143774    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bf4a50c-471b-4019-bff6-404c1ce9fbce-metrics-certs\") pod \"maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97\" (UID: \"3bf4a50c-471b-4019-bff6-404c1ce9fbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97"
Oct 30 01:33:30.144042 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.143832    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7thx6\" (UniqueName: \"kubernetes.io/projected/3bf4a50c-471b-4019-bff6-404c1ce9fbce-kube-api-access-7thx6\") pod \"maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97\" (UID: \"3bf4a50c-471b-4019-bff6-404c1ce9fbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97"
Oct 30 01:33:30.144537 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.144514    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf4a50c-471b-4019-bff6-404c1ce9fbce-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97\" (UID: \"3bf4a50c-471b-4019-bff6-404c1ce9fbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97"
Oct 30 01:33:30.146639 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.146610    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bf4a50c-471b-4019-bff6-404c1ce9fbce-metrics-certs\") pod \"maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97\" (UID: \"3bf4a50c-471b-4019-bff6-404c1ce9fbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97"
Oct 30 01:33:30.146768 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.146741    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bf4a50c-471b-4019-bff6-404c1ce9fbce-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97\" (UID: \"3bf4a50c-471b-4019-bff6-404c1ce9fbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97"
Oct 30 01:33:30.169184 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.169146    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7thx6\" (UniqueName: \"kubernetes.io/projected/3bf4a50c-471b-4019-bff6-404c1ce9fbce-kube-api-access-7thx6\") pod \"maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97\" (UID: \"3bf4a50c-471b-4019-bff6-404c1ce9fbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97"
Oct 30 01:33:30.236358 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.236257    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe8356-57d4bpvtqx"]
Oct 30 01:33:30.240362 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.240334    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe8356-57d4bpvtqx"
Oct 30 01:33:30.259748 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.259710    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-4d76e3c2-dockercfg-hrwbm\""
Oct 30 01:33:30.260064 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.260027    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-fe8356\""
Oct 30 01:33:30.262912 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.262883    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe8356-57d4bpvtqx"]
Oct 30 01:33:30.272881 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.272838    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97"
Oct 30 01:33:30.297273 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.297228    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fe8356-79cd4c697vj9c4"]
Oct 30 01:33:30.302370 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.302343    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe8356-79cd4c697vj9c4"
Oct 30 01:33:30.328197 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.327177    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fe8356-79cd4c697vj9c4"]
Oct 30 01:33:30.346822 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.345634    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q9bg\" (UniqueName: \"kubernetes.io/projected/e90e9c5f-6d82-4b96-a4c1-f729e0f33080-kube-api-access-9q9bg\") pod \"payload-pre-processing-e2e-worker-single-main-fe8356-57d4bpvtqx\" (UID: \"e90e9c5f-6d82-4b96-a4c1-f729e0f33080\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe8356-57d4bpvtqx"
Oct 30 01:33:30.346822 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.345706    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42rb9\" (UniqueName: \"kubernetes.io/projected/126a90f2-6867-4d83-8b4f-89e1939b492c-kube-api-access-42rb9\") pod \"payload-processing-e2e-worker-single-main-fe8356-79cd4c697vj9c4\" (UID: \"126a90f2-6867-4d83-8b4f-89e1939b492c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe8356-79cd4c697vj9c4"
Oct 30 01:33:30.346822 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.345746    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e90e9c5f-6d82-4b96-a4c1-f729e0f33080-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-fe8356-57d4bpvtqx\" (UID: \"e90e9c5f-6d82-4b96-a4c1-f729e0f33080\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe8356-57d4bpvtqx"
Oct 30 01:33:30.359844 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.349194    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/126a90f2-6867-4d83-8b4f-89e1939b492c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-fe8356-79cd4c697vj9c4\" (UID: \"126a90f2-6867-4d83-8b4f-89e1939b492c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe8356-79cd4c697vj9c4"
Oct 30 01:33:30.450877 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.450829    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q9bg\" (UniqueName: \"kubernetes.io/projected/e90e9c5f-6d82-4b96-a4c1-f729e0f33080-kube-api-access-9q9bg\") pod \"payload-pre-processing-e2e-worker-single-main-fe8356-57d4bpvtqx\" (UID: \"e90e9c5f-6d82-4b96-a4c1-f729e0f33080\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe8356-57d4bpvtqx"
Oct 30 01:33:30.451096 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.450894    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42rb9\" (UniqueName: \"kubernetes.io/projected/126a90f2-6867-4d83-8b4f-89e1939b492c-kube-api-access-42rb9\") pod \"payload-processing-e2e-worker-single-main-fe8356-79cd4c697vj9c4\" (UID: \"126a90f2-6867-4d83-8b4f-89e1939b492c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe8356-79cd4c697vj9c4"
Oct 30 01:33:30.451096 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.450925    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e90e9c5f-6d82-4b96-a4c1-f729e0f33080-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-fe8356-57d4bpvtqx\" (UID: \"e90e9c5f-6d82-4b96-a4c1-f729e0f33080\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe8356-57d4bpvtqx"
Oct 30 01:33:30.451096 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.451053    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/126a90f2-6867-4d83-8b4f-89e1939b492c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-fe8356-79cd4c697vj9c4\" (UID: \"126a90f2-6867-4d83-8b4f-89e1939b492c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe8356-79cd4c697vj9c4"
Oct 30 01:33:30.451813 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.451774    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e90e9c5f-6d82-4b96-a4c1-f729e0f33080-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-fe8356-57d4bpvtqx\" (UID: \"e90e9c5f-6d82-4b96-a4c1-f729e0f33080\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe8356-57d4bpvtqx"
Oct 30 01:33:30.451856 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.451795    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/126a90f2-6867-4d83-8b4f-89e1939b492c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-fe8356-79cd4c697vj9c4\" (UID: \"126a90f2-6867-4d83-8b4f-89e1939b492c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe8356-79cd4c697vj9c4"
Oct 30 01:33:30.473856 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.473821    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42rb9\" (UniqueName: \"kubernetes.io/projected/126a90f2-6867-4d83-8b4f-89e1939b492c-kube-api-access-42rb9\") pod \"payload-processing-e2e-worker-single-main-fe8356-79cd4c697vj9c4\" (UID: \"126a90f2-6867-4d83-8b4f-89e1939b492c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe8356-79cd4c697vj9c4"
Oct 30 01:33:30.473856 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.473851    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q9bg\" (UniqueName: \"kubernetes.io/projected/e90e9c5f-6d82-4b96-a4c1-f729e0f33080-kube-api-access-9q9bg\") pod \"payload-pre-processing-e2e-worker-single-main-fe8356-57d4bpvtqx\" (UID: \"e90e9c5f-6d82-4b96-a4c1-f729e0f33080\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe8356-57d4bpvtqx"
Oct 30 01:33:30.555573 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.555477    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe8356-57d4bpvtqx"
Oct 30 01:33:30.626530 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:33:30.626472    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf4a50c_471b_4019_bff6_404c1ce9fbce.slice/crio-e561b45c1e682cb618d980c11b5bb3eda9d37a5b51b17ba4a8776250a8591122 WatchSource:0}: Error finding container e561b45c1e682cb618d980c11b5bb3eda9d37a5b51b17ba4a8776250a8591122: Status 404 returned error can't find the container with id e561b45c1e682cb618d980c11b5bb3eda9d37a5b51b17ba4a8776250a8591122
Oct 30 01:33:30.629155 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.629124    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97"]
Oct 30 01:33:30.667684 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.667638    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe8356-79cd4c697vj9c4"
Oct 30 01:33:30.875151 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.873083    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe8356-57d4bpvtqx"]
Oct 30 01:33:30.884533 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:33:30.884490    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode90e9c5f_6d82_4b96_a4c1_f729e0f33080.slice/crio-573a1455cd82207c0b3c74c269bfd379606fe5af553f34f9a1e84b2acef959c5 WatchSource:0}: Error finding container 573a1455cd82207c0b3c74c269bfd379606fe5af553f34f9a1e84b2acef959c5: Status 404 returned error can't find the container with id 573a1455cd82207c0b3c74c269bfd379606fe5af553f34f9a1e84b2acef959c5
Oct 30 01:33:30.974206 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:30.974174    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fe8356-79cd4c697vj9c4"]
Oct 30 01:33:30.975746 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:33:30.975707    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod126a90f2_6867_4d83_8b4f_89e1939b492c.slice/crio-ad0936c546d543b0b1b550b7cc086a956bcedacc3d65a2776c62c6653be1d328 WatchSource:0}: Error finding container ad0936c546d543b0b1b550b7cc086a956bcedacc3d65a2776c62c6653be1d328: Status 404 returned error can't find the container with id ad0936c546d543b0b1b550b7cc086a956bcedacc3d65a2776c62c6653be1d328
Oct 30 01:33:31.493660 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:31.493622    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97" event={"ID":"3bf4a50c-471b-4019-bff6-404c1ce9fbce","Type":"ContainerStarted","Data":"e833fc1a512bb3a9602aaa04ea3d1c5772be617f1f501ca4a7ef66b5a24eac3a"}
Oct 30 01:33:31.493953 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:31.493679    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97" event={"ID":"3bf4a50c-471b-4019-bff6-404c1ce9fbce","Type":"ContainerStarted","Data":"e561b45c1e682cb618d980c11b5bb3eda9d37a5b51b17ba4a8776250a8591122"}
Oct 30 01:33:31.493953 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:31.493906    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97"
Oct 30 01:33:31.495683 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:31.495654    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe8356-79cd4c697vj9c4" event={"ID":"126a90f2-6867-4d83-8b4f-89e1939b492c","Type":"ContainerStarted","Data":"a488bb199ac4687841fd951c1fc9f04b0e2653fb50bd8c95c13591d112c2dce1"}
Oct 30 01:33:31.495880 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:31.495700    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe8356-79cd4c697vj9c4" event={"ID":"126a90f2-6867-4d83-8b4f-89e1939b492c","Type":"ContainerStarted","Data":"ad0936c546d543b0b1b550b7cc086a956bcedacc3d65a2776c62c6653be1d328"}
Oct 30 01:33:31.495880 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:31.495745    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe8356-79cd4c697vj9c4"
Oct 30 01:33:31.497454 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:31.497427    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe8356-57d4bpvtqx" event={"ID":"e90e9c5f-6d82-4b96-a4c1-f729e0f33080","Type":"ContainerStarted","Data":"09a2b33392ff16d73b30c961fd93843b64adbc515c5f831d28f4ede0cab2970c"}
Oct 30 01:33:31.497577 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:31.497475    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe8356-57d4bpvtqx" event={"ID":"e90e9c5f-6d82-4b96-a4c1-f729e0f33080","Type":"ContainerStarted","Data":"573a1455cd82207c0b3c74c269bfd379606fe5af553f34f9a1e84b2acef959c5"}
Oct 30 01:33:31.497577 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:31.497559    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe8356-57d4bpvtqx"
Oct 30 01:33:31.537135 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:31.537078    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97" podStartSLOduration=2.177686617 podStartE2EDuration="2.537063908s" totalImagesPullingTime="359.377291ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:31.535248553 +0000 UTC m=+3701.922924943" watchObservedRunningTime="2026-10-30 01:33:31.537063908 +0000 UTC m=+3701.924740301"
Oct 30 01:33:31.585673 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:31.585613    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe8356-79cd4c697vj9c4" podStartSLOduration=1.5855997469999998 podStartE2EDuration="1.585599747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:31.584241766 +0000 UTC m=+3701.971918158" watchObservedRunningTime="2026-10-30 01:33:31.585599747 +0000 UTC m=+3701.973276193"
Oct 30 01:33:31.624604 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:31.624538    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe8356-57d4bpvtqx" podStartSLOduration=1.6245175010000001 podStartE2EDuration="1.624517501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:31.623117668 +0000 UTC m=+3702.010794055" watchObservedRunningTime="2026-10-30 01:33:31.624517501 +0000 UTC m=+3702.012193891"
Oct 30 01:33:37.510155 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:37.510118    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97"
Oct 30 01:33:42.507389 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:42.507358    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe8356-57d4bpvtqx"
Oct 30 01:33:42.507842 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:42.507590    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe8356-79cd4c697vj9c4"
Oct 30 01:33:45.273137 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:45.273101    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv"]
Oct 30 01:33:45.278117 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:45.278096    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv"
Oct 30 01:33:45.294308 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:45.294281    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-fe8356\"/\"openshift-service-ca.crt\""
Oct 30 01:33:45.305893 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:45.305861    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-fe8356\"/\"kube-root-ca.crt\""
Oct 30 01:33:45.306048 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:45.305982    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-fe8356\"/\"simulator-single-main-fe8356-kserve-self-signed-certs\""
Oct 30 01:33:45.306048 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:45.306007    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-fe8356\"/\"default-dockercfg-jnsqd\""
Oct 30 01:33:45.308865 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:45.308842    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv"]
Oct 30 01:33:45.414968 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:45.414937    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2936212f-2a9d-4345-a246-1a89b0af3114-dshm\") pod \"simulator-single-main-fe8356-kserve-594975c665-tgshv\" (UID: \"2936212f-2a9d-4345-a246-1a89b0af3114\") " pod="e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv"
Oct 30 01:33:45.415161 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:45.414983    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2936212f-2a9d-4345-a246-1a89b0af3114-home\") pod \"simulator-single-main-fe8356-kserve-594975c665-tgshv\" (UID: \"2936212f-2a9d-4345-a246-1a89b0af3114\") " pod="e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv"
Oct 30 01:33:45.415161 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:45.415067    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2936212f-2a9d-4345-a246-1a89b0af3114-tmp-dir\") pod \"simulator-single-main-fe8356-kserve-594975c665-tgshv\" (UID: \"2936212f-2a9d-4345-a246-1a89b0af3114\") " pod="e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv"
Oct 30 01:33:45.415161 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:45.415105    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2936212f-2a9d-4345-a246-1a89b0af3114-model-cache\") pod \"simulator-single-main-fe8356-kserve-594975c665-tgshv\" (UID: \"2936212f-2a9d-4345-a246-1a89b0af3114\") " pod="e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv"
Oct 30 01:33:45.415296 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:45.415223    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2936212f-2a9d-4345-a246-1a89b0af3114-tls-certs\") pod \"simulator-single-main-fe8356-kserve-594975c665-tgshv\" (UID: \"2936212f-2a9d-4345-a246-1a89b0af3114\") " pod="e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv"
Oct 30 01:33:45.415296 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:45.415265    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27jb2\" (UniqueName: \"kubernetes.io/projected/2936212f-2a9d-4345-a246-1a89b0af3114-kube-api-access-27jb2\") pod \"simulator-single-main-fe8356-kserve-594975c665-tgshv\" (UID: \"2936212f-2a9d-4345-a246-1a89b0af3114\") " pod="e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv"
Oct 30 01:33:45.516425 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:45.516395    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27jb2\" (UniqueName: \"kubernetes.io/projected/2936212f-2a9d-4345-a246-1a89b0af3114-kube-api-access-27jb2\") pod \"simulator-single-main-fe8356-kserve-594975c665-tgshv\" (UID: \"2936212f-2a9d-4345-a246-1a89b0af3114\") " pod="e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv"
Oct 30 01:33:45.516601 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:45.516435    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2936212f-2a9d-4345-a246-1a89b0af3114-dshm\") pod \"simulator-single-main-fe8356-kserve-594975c665-tgshv\" (UID: \"2936212f-2a9d-4345-a246-1a89b0af3114\") " pod="e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv"
Oct 30 01:33:45.516601 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:45.516464    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2936212f-2a9d-4345-a246-1a89b0af3114-home\") pod \"simulator-single-main-fe8356-kserve-594975c665-tgshv\" (UID: \"2936212f-2a9d-4345-a246-1a89b0af3114\") " pod="e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv"
Oct 30 01:33:45.516601 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:45.516484    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2936212f-2a9d-4345-a246-1a89b0af3114-tmp-dir\") pod \"simulator-single-main-fe8356-kserve-594975c665-tgshv\" (UID: \"2936212f-2a9d-4345-a246-1a89b0af3114\") " pod="e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv"
Oct 30 01:33:45.516601 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:45.516520    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2936212f-2a9d-4345-a246-1a89b0af3114-model-cache\") pod \"simulator-single-main-fe8356-kserve-594975c665-tgshv\" (UID: \"2936212f-2a9d-4345-a246-1a89b0af3114\") " pod="e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv"
Oct 30 01:33:45.516756 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:45.516605    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2936212f-2a9d-4345-a246-1a89b0af3114-tls-certs\") pod \"simulator-single-main-fe8356-kserve-594975c665-tgshv\" (UID: \"2936212f-2a9d-4345-a246-1a89b0af3114\") " pod="e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv"
Oct 30 01:33:45.516956 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:45.516921    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2936212f-2a9d-4345-a246-1a89b0af3114-home\") pod \"simulator-single-main-fe8356-kserve-594975c665-tgshv\" (UID: \"2936212f-2a9d-4345-a246-1a89b0af3114\") " pod="e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv"
Oct 30 01:33:45.517080 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:45.516995    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2936212f-2a9d-4345-a246-1a89b0af3114-tmp-dir\") pod \"simulator-single-main-fe8356-kserve-594975c665-tgshv\" (UID: \"2936212f-2a9d-4345-a246-1a89b0af3114\") " pod="e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv"
Oct 30 01:33:45.517080 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:45.517027    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2936212f-2a9d-4345-a246-1a89b0af3114-model-cache\") pod \"simulator-single-main-fe8356-kserve-594975c665-tgshv\" (UID: \"2936212f-2a9d-4345-a246-1a89b0af3114\") " pod="e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv"
Oct 30 01:33:45.518957 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:45.518932    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2936212f-2a9d-4345-a246-1a89b0af3114-dshm\") pod \"simulator-single-main-fe8356-kserve-594975c665-tgshv\" (UID: \"2936212f-2a9d-4345-a246-1a89b0af3114\") " pod="e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv"
Oct 30 01:33:45.519270 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:45.519255    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2936212f-2a9d-4345-a246-1a89b0af3114-tls-certs\") pod \"simulator-single-main-fe8356-kserve-594975c665-tgshv\" (UID: \"2936212f-2a9d-4345-a246-1a89b0af3114\") " pod="e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv"
Oct 30 01:33:45.548344 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:45.548273    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27jb2\" (UniqueName: \"kubernetes.io/projected/2936212f-2a9d-4345-a246-1a89b0af3114-kube-api-access-27jb2\") pod \"simulator-single-main-fe8356-kserve-594975c665-tgshv\" (UID: \"2936212f-2a9d-4345-a246-1a89b0af3114\") " pod="e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv"
Oct 30 01:33:45.591947 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:45.591915    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv"
Oct 30 01:33:45.899622 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:45.899593    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv"]
Oct 30 01:33:45.899942 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:33:45.899916    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2936212f_2a9d_4345_a246_1a89b0af3114.slice/crio-bd12f256e55a06f1e964acafccd1dccfbf60ac93fe9342cffa937d858ba94ae5 WatchSource:0}: Error finding container bd12f256e55a06f1e964acafccd1dccfbf60ac93fe9342cffa937d858ba94ae5: Status 404 returned error can't find the container with id bd12f256e55a06f1e964acafccd1dccfbf60ac93fe9342cffa937d858ba94ae5
Oct 30 01:33:46.596145 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:46.596105    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv" event={"ID":"2936212f-2a9d-4345-a246-1a89b0af3114","Type":"ContainerStarted","Data":"bdeb41699f52c5c8dbf57724470c5516bae2b05cd9158e53ef7e1942080b2131"}
Oct 30 01:33:46.596145 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:46.596146    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv" event={"ID":"2936212f-2a9d-4345-a246-1a89b0af3114","Type":"ContainerStarted","Data":"bd12f256e55a06f1e964acafccd1dccfbf60ac93fe9342cffa937d858ba94ae5"}
Oct 30 01:33:46.636448 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:46.636392    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv" podStartSLOduration=1.636378782 podStartE2EDuration="1.636378782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:46.635165385 +0000 UTC m=+3717.022841774" watchObservedRunningTime="2026-10-30 01:33:46.636378782 +0000 UTC m=+3717.024055171"
Oct 30 01:33:47.592823 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:47.592760    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv"
Oct 30 01:33:55.592557 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:55.592511    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv"
Oct 30 01:33:55.605560 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:55.605528    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv"
Oct 30 01:33:55.661069 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:33:55.661040    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv"
Oct 30 01:33:56.511230 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:33:56.511188    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:34:20.508153 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:20.508122    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv"]
Oct 30 01:34:26.062360 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:26.062305    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fe8356-79cd4c697vj9c4"]
Oct 30 01:34:26.091646 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:26.091612    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe8356-57d4bpvtqx"]
Oct 30 01:34:26.510510 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:34:26.510472    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:34:28.573683 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:28.573647    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97"]
Oct 30 01:34:28.574155 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:28.573870    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97" podUID="3bf4a50c-471b-4019-bff6-404c1ce9fbce" containerName="maas-api" containerID="cri-o://e833fc1a512bb3a9602aaa04ea3d1c5772be617f1f501ca4a7ef66b5a24eac3a" gracePeriod=30
Oct 30 01:34:29.825209 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:29.825183    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97"
Oct 30 01:34:29.846267 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:29.846234    4393 generic.go:356] "Generic (PLEG): container finished" podID="3bf4a50c-471b-4019-bff6-404c1ce9fbce" containerID="e833fc1a512bb3a9602aaa04ea3d1c5772be617f1f501ca4a7ef66b5a24eac3a" exitCode=0
Oct 30 01:34:29.846460 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:29.846320    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97"
Oct 30 01:34:29.846460 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:29.846350    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97" event={"ID":"3bf4a50c-471b-4019-bff6-404c1ce9fbce","Type":"ContainerDied","Data":"e833fc1a512bb3a9602aaa04ea3d1c5772be617f1f501ca4a7ef66b5a24eac3a"}
Oct 30 01:34:29.846460 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:29.846406    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97" event={"ID":"3bf4a50c-471b-4019-bff6-404c1ce9fbce","Type":"ContainerDied","Data":"e561b45c1e682cb618d980c11b5bb3eda9d37a5b51b17ba4a8776250a8591122"}
Oct 30 01:34:29.846460 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:29.846428    4393 scope.go:122] "RemoveContainer" containerID="e833fc1a512bb3a9602aaa04ea3d1c5772be617f1f501ca4a7ef66b5a24eac3a"
Oct 30 01:34:29.856646 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:29.856622    4393 scope.go:122] "RemoveContainer" containerID="e833fc1a512bb3a9602aaa04ea3d1c5772be617f1f501ca4a7ef66b5a24eac3a"
Oct 30 01:34:29.857037 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:34:29.857013    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e833fc1a512bb3a9602aaa04ea3d1c5772be617f1f501ca4a7ef66b5a24eac3a\": container with ID starting with e833fc1a512bb3a9602aaa04ea3d1c5772be617f1f501ca4a7ef66b5a24eac3a not found: ID does not exist" containerID="e833fc1a512bb3a9602aaa04ea3d1c5772be617f1f501ca4a7ef66b5a24eac3a"
Oct 30 01:34:29.857128 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:29.857048    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e833fc1a512bb3a9602aaa04ea3d1c5772be617f1f501ca4a7ef66b5a24eac3a"} err="failed to get container status \"e833fc1a512bb3a9602aaa04ea3d1c5772be617f1f501ca4a7ef66b5a24eac3a\": rpc error: code = NotFound desc = could not find container \"e833fc1a512bb3a9602aaa04ea3d1c5772be617f1f501ca4a7ef66b5a24eac3a\": container with ID starting with e833fc1a512bb3a9602aaa04ea3d1c5772be617f1f501ca4a7ef66b5a24eac3a not found: ID does not exist"
Oct 30 01:34:29.874370 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:29.874329    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bf4a50c-471b-4019-bff6-404c1ce9fbce-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf4a50c-471b-4019-bff6-404c1ce9fbce-openshift-service-ca\") pod \"3bf4a50c-471b-4019-bff6-404c1ce9fbce\" (UID: \"3bf4a50c-471b-4019-bff6-404c1ce9fbce\") "
Oct 30 01:34:29.874554 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:29.874453    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bf4a50c-471b-4019-bff6-404c1ce9fbce-kube-api-access-7thx6\" (UniqueName: \"kubernetes.io/projected/3bf4a50c-471b-4019-bff6-404c1ce9fbce-kube-api-access-7thx6\") pod \"3bf4a50c-471b-4019-bff6-404c1ce9fbce\" (UID: \"3bf4a50c-471b-4019-bff6-404c1ce9fbce\") "
Oct 30 01:34:29.874554 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:29.874508    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bf4a50c-471b-4019-bff6-404c1ce9fbce-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bf4a50c-471b-4019-bff6-404c1ce9fbce-metrics-certs\") pod \"3bf4a50c-471b-4019-bff6-404c1ce9fbce\" (UID: \"3bf4a50c-471b-4019-bff6-404c1ce9fbce\") "
Oct 30 01:34:29.874683 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:29.874608    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bf4a50c-471b-4019-bff6-404c1ce9fbce-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bf4a50c-471b-4019-bff6-404c1ce9fbce-maas-api-tls\") pod \"3bf4a50c-471b-4019-bff6-404c1ce9fbce\" (UID: \"3bf4a50c-471b-4019-bff6-404c1ce9fbce\") "
Oct 30 01:34:29.875264 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:29.875215    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf4a50c-471b-4019-bff6-404c1ce9fbce-openshift-service-ca" pod "3bf4a50c-471b-4019-bff6-404c1ce9fbce" (UID: "3bf4a50c-471b-4019-bff6-404c1ce9fbce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:34:29.877660 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:29.877630    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bf4a50c-471b-4019-bff6-404c1ce9fbce-metrics-certs" pod "3bf4a50c-471b-4019-bff6-404c1ce9fbce" (UID: "3bf4a50c-471b-4019-bff6-404c1ce9fbce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:34:29.878012 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:29.877987    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bf4a50c-471b-4019-bff6-404c1ce9fbce-maas-api-tls" pod "3bf4a50c-471b-4019-bff6-404c1ce9fbce" (UID: "3bf4a50c-471b-4019-bff6-404c1ce9fbce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:34:29.878183 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:29.878110    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf4a50c-471b-4019-bff6-404c1ce9fbce-kube-api-access-7thx6" pod "3bf4a50c-471b-4019-bff6-404c1ce9fbce" (UID: "3bf4a50c-471b-4019-bff6-404c1ce9fbce"). InnerVolumeSpecName "kube-api-access-7thx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:34:29.976807 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:29.976684    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7thx6\" (UniqueName: \"kubernetes.io/projected/3bf4a50c-471b-4019-bff6-404c1ce9fbce-kube-api-access-7thx6\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:34:29.976807 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:29.976729    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bf4a50c-471b-4019-bff6-404c1ce9fbce-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:34:29.976807 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:29.976744    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bf4a50c-471b-4019-bff6-404c1ce9fbce-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:34:29.976807 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:29.976757    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bf4a50c-471b-4019-bff6-404c1ce9fbce-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:34:30.191347 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:30.191310    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97"]
Oct 30 01:34:30.202694 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:30.202664    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-fe8356-7f47448cc8-whb97"]
Oct 30 01:34:31.063088 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:31.063039    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe8356-79cd4c697vj9c4" podUID="126a90f2-6867-4d83-8b4f-89e1939b492c" containerName="payload-processing" containerID="cri-o://a488bb199ac4687841fd951c1fc9f04b0e2653fb50bd8c95c13591d112c2dce1" gracePeriod=25
Oct 30 01:34:31.092941 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:31.092880    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe8356-57d4bpvtqx" podUID="e90e9c5f-6d82-4b96-a4c1-f729e0f33080" containerName="payload-pre-processing" containerID="cri-o://09a2b33392ff16d73b30c961fd93843b64adbc515c5f831d28f4ede0cab2970c" gracePeriod=25
Oct 30 01:34:31.724480 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:31.724443    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bf4a50c-471b-4019-bff6-404c1ce9fbce" path="/var/lib/kubelet/pods/3bf4a50c-471b-4019-bff6-404c1ce9fbce/volumes"
Oct 30 01:34:32.505498 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:32.505457    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe8356-57d4bpvtqx" podUID="e90e9c5f-6d82-4b96-a4c1-f729e0f33080" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:34:32.506003 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:32.505472    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe8356-79cd4c697vj9c4" podUID="126a90f2-6867-4d83-8b4f-89e1939b492c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:34:35.529162 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:35.529123    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv" podUID="2936212f-2a9d-4345-a246-1a89b0af3114" containerName="main" containerID="cri-o://bdeb41699f52c5c8dbf57724470c5516bae2b05cd9158e53ef7e1942080b2131" gracePeriod=45
Oct 30 01:34:35.650229 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:35.650192    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv" podUID="2936212f-2a9d-4345-a246-1a89b0af3114" containerName="main" probeResult="failure" output="Get \"https://10.133.0.210:8000/ready\": dial tcp 10.133.0.210:8000: connect: connection refused"
Oct 30 01:34:35.774291 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:35.774264    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv"
Oct 30 01:34:35.838134 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:35.838026    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2936212f-2a9d-4345-a246-1a89b0af3114-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2936212f-2a9d-4345-a246-1a89b0af3114-tmp-dir\") pod \"2936212f-2a9d-4345-a246-1a89b0af3114\" (UID: \"2936212f-2a9d-4345-a246-1a89b0af3114\") "
Oct 30 01:34:35.838341 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:35.838195    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2936212f-2a9d-4345-a246-1a89b0af3114-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2936212f-2a9d-4345-a246-1a89b0af3114-model-cache\") pod \"2936212f-2a9d-4345-a246-1a89b0af3114\" (UID: \"2936212f-2a9d-4345-a246-1a89b0af3114\") "
Oct 30 01:34:35.838341 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:35.838226    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2936212f-2a9d-4345-a246-1a89b0af3114-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2936212f-2a9d-4345-a246-1a89b0af3114-dshm\") pod \"2936212f-2a9d-4345-a246-1a89b0af3114\" (UID: \"2936212f-2a9d-4345-a246-1a89b0af3114\") "
Oct 30 01:34:35.838341 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:35.838258    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2936212f-2a9d-4345-a246-1a89b0af3114-home\" (UniqueName: \"kubernetes.io/empty-dir/2936212f-2a9d-4345-a246-1a89b0af3114-home\") pod \"2936212f-2a9d-4345-a246-1a89b0af3114\" (UID: \"2936212f-2a9d-4345-a246-1a89b0af3114\") "
Oct 30 01:34:35.838341 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:35.838310    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2936212f-2a9d-4345-a246-1a89b0af3114-kube-api-access-27jb2\" (UniqueName: \"kubernetes.io/projected/2936212f-2a9d-4345-a246-1a89b0af3114-kube-api-access-27jb2\") pod \"2936212f-2a9d-4345-a246-1a89b0af3114\" (UID: \"2936212f-2a9d-4345-a246-1a89b0af3114\") "
Oct 30 01:34:35.838341 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:35.838315    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2936212f-2a9d-4345-a246-1a89b0af3114-tmp-dir" pod "2936212f-2a9d-4345-a246-1a89b0af3114" (UID: "2936212f-2a9d-4345-a246-1a89b0af3114"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:34:35.838658 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:35.838352    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2936212f-2a9d-4345-a246-1a89b0af3114-tls-certs\" (UniqueName: \"kubernetes.io/secret/2936212f-2a9d-4345-a246-1a89b0af3114-tls-certs\") pod \"2936212f-2a9d-4345-a246-1a89b0af3114\" (UID: \"2936212f-2a9d-4345-a246-1a89b0af3114\") "
Oct 30 01:34:35.838658 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:35.838526    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2936212f-2a9d-4345-a246-1a89b0af3114-model-cache" pod "2936212f-2a9d-4345-a246-1a89b0af3114" (UID: "2936212f-2a9d-4345-a246-1a89b0af3114"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:34:35.838658 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:35.838548    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2936212f-2a9d-4345-a246-1a89b0af3114-home" pod "2936212f-2a9d-4345-a246-1a89b0af3114" (UID: "2936212f-2a9d-4345-a246-1a89b0af3114"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:34:35.838857 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:35.838732    4393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2936212f-2a9d-4345-a246-1a89b0af3114-tmp-dir\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:34:35.838857 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:35.838754    4393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2936212f-2a9d-4345-a246-1a89b0af3114-model-cache\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:34:35.838857 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:35.838770    4393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2936212f-2a9d-4345-a246-1a89b0af3114-home\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:34:35.841117 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:35.841095    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2936212f-2a9d-4345-a246-1a89b0af3114-tls-certs" pod "2936212f-2a9d-4345-a246-1a89b0af3114" (UID: "2936212f-2a9d-4345-a246-1a89b0af3114"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:34:35.841322 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:35.841305    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2936212f-2a9d-4345-a246-1a89b0af3114-kube-api-access-27jb2" pod "2936212f-2a9d-4345-a246-1a89b0af3114" (UID: "2936212f-2a9d-4345-a246-1a89b0af3114"). InnerVolumeSpecName "kube-api-access-27jb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:34:35.841464 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:35.841439    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2936212f-2a9d-4345-a246-1a89b0af3114-dshm" pod "2936212f-2a9d-4345-a246-1a89b0af3114" (UID: "2936212f-2a9d-4345-a246-1a89b0af3114"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:34:35.885469 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:35.885435    4393 generic.go:356] "Generic (PLEG): container finished" podID="2936212f-2a9d-4345-a246-1a89b0af3114" containerID="bdeb41699f52c5c8dbf57724470c5516bae2b05cd9158e53ef7e1942080b2131" exitCode=0
Oct 30 01:34:35.885666 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:35.885494    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv"
Oct 30 01:34:35.885666 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:35.885516    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv" event={"ID":"2936212f-2a9d-4345-a246-1a89b0af3114","Type":"ContainerDied","Data":"bdeb41699f52c5c8dbf57724470c5516bae2b05cd9158e53ef7e1942080b2131"}
Oct 30 01:34:35.885666 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:35.885570    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv" event={"ID":"2936212f-2a9d-4345-a246-1a89b0af3114","Type":"ContainerDied","Data":"bd12f256e55a06f1e964acafccd1dccfbf60ac93fe9342cffa937d858ba94ae5"}
Oct 30 01:34:35.885666 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:35.885592    4393 scope.go:122] "RemoveContainer" containerID="bdeb41699f52c5c8dbf57724470c5516bae2b05cd9158e53ef7e1942080b2131"
Oct 30 01:34:35.896294 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:35.896279    4393 scope.go:122] "RemoveContainer" containerID="bdeb41699f52c5c8dbf57724470c5516bae2b05cd9158e53ef7e1942080b2131"
Oct 30 01:34:35.896589 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:34:35.896570    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdeb41699f52c5c8dbf57724470c5516bae2b05cd9158e53ef7e1942080b2131\": container with ID starting with bdeb41699f52c5c8dbf57724470c5516bae2b05cd9158e53ef7e1942080b2131 not found: ID does not exist" containerID="bdeb41699f52c5c8dbf57724470c5516bae2b05cd9158e53ef7e1942080b2131"
Oct 30 01:34:35.896656 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:35.896600    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdeb41699f52c5c8dbf57724470c5516bae2b05cd9158e53ef7e1942080b2131"} err="failed to get container status \"bdeb41699f52c5c8dbf57724470c5516bae2b05cd9158e53ef7e1942080b2131\": rpc error: code = NotFound desc = could not find container \"bdeb41699f52c5c8dbf57724470c5516bae2b05cd9158e53ef7e1942080b2131\": container with ID starting with bdeb41699f52c5c8dbf57724470c5516bae2b05cd9158e53ef7e1942080b2131 not found: ID does not exist"
Oct 30 01:34:35.936512 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:35.936482    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv"]
Oct 30 01:34:35.939938 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:35.939918    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-fe8356/simulator-single-main-fe8356-kserve-594975c665-tgshv"]
Oct 30 01:34:35.940076 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:35.940060    4393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2936212f-2a9d-4345-a246-1a89b0af3114-dshm\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:34:35.940133 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:35.940080    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27jb2\" (UniqueName: \"kubernetes.io/projected/2936212f-2a9d-4345-a246-1a89b0af3114-kube-api-access-27jb2\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:34:35.940133 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:35.940090    4393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2936212f-2a9d-4345-a246-1a89b0af3114-tls-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:34:37.724486 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:37.724450    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2936212f-2a9d-4345-a246-1a89b0af3114" path="/var/lib/kubelet/pods/2936212f-2a9d-4345-a246-1a89b0af3114/volumes"
Oct 30 01:34:41.374928 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:41.374888    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh"]
Oct 30 01:34:41.375639 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:41.375617    4393 cpu_manager.go:414] "containerMap: removing container" podUID="3bf4a50c-471b-4019-bff6-404c1ce9fbce" containerName="maas-api"
Oct 30 01:34:41.375686 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:41.375644    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf4a50c-471b-4019-bff6-404c1ce9fbce" containerName="maas-api"
Oct 30 01:34:41.375686 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:41.375664    4393 cpu_manager.go:414] "containerMap: removing container" podUID="2936212f-2a9d-4345-a246-1a89b0af3114" containerName="main"
Oct 30 01:34:41.375686 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:41.375674    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2936212f-2a9d-4345-a246-1a89b0af3114" containerName="main"
Oct 30 01:34:41.375857 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:41.375842    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf4a50c-471b-4019-bff6-404c1ce9fbce" containerName="maas-api"
Oct 30 01:34:41.375901 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:41.375862    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="2936212f-2a9d-4345-a246-1a89b0af3114" containerName="main"
Oct 30 01:34:41.380846 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:41.380822    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh"
Oct 30 01:34:41.397200 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:41.397161    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh"]
Oct 30 01:34:41.406022 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:41.405987    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-e28ec8a1-kserve-self-signed-certs\""
Oct 30 01:34:41.496438 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:41.496402    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-tmp-dir\") pod \"e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh\" (UID: \"819a86e9-fe42-4f5b-ae6c-944d2a20a80c\") " pod="llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh"
Oct 30 01:34:41.496631 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:41.496467    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-home\") pod \"e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh\" (UID: \"819a86e9-fe42-4f5b-ae6c-944d2a20a80c\") " pod="llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh"
Oct 30 01:34:41.496631 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:41.496604    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7p77\" (UniqueName: \"kubernetes.io/projected/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-kube-api-access-x7p77\") pod \"e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh\" (UID: \"819a86e9-fe42-4f5b-ae6c-944d2a20a80c\") " pod="llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh"
Oct 30 01:34:41.496828 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:41.496661    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-model-cache\") pod \"e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh\" (UID: \"819a86e9-fe42-4f5b-ae6c-944d2a20a80c\") " pod="llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh"
Oct 30 01:34:41.496828 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:41.496769    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-dshm\") pod \"e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh\" (UID: \"819a86e9-fe42-4f5b-ae6c-944d2a20a80c\") " pod="llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh"
Oct 30 01:34:41.496927 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:41.496832    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-tls-certs\") pod \"e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh\" (UID: \"819a86e9-fe42-4f5b-ae6c-944d2a20a80c\") " pod="llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh"
Oct 30 01:34:41.597844 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:41.597805    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7p77\" (UniqueName: \"kubernetes.io/projected/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-kube-api-access-x7p77\") pod \"e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh\" (UID: \"819a86e9-fe42-4f5b-ae6c-944d2a20a80c\") " pod="llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh"
Oct 30 01:34:41.598021 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:41.597876    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-model-cache\") pod \"e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh\" (UID: \"819a86e9-fe42-4f5b-ae6c-944d2a20a80c\") " pod="llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh"
Oct 30 01:34:41.598021 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:41.597931    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-dshm\") pod \"e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh\" (UID: \"819a86e9-fe42-4f5b-ae6c-944d2a20a80c\") " pod="llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh"
Oct 30 01:34:41.598021 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:41.597957    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-tls-certs\") pod \"e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh\" (UID: \"819a86e9-fe42-4f5b-ae6c-944d2a20a80c\") " pod="llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh"
Oct 30 01:34:41.598021 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:41.598015    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-tmp-dir\") pod \"e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh\" (UID: \"819a86e9-fe42-4f5b-ae6c-944d2a20a80c\") " pod="llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh"
Oct 30 01:34:41.598165 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:41.598099    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-home\") pod \"e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh\" (UID: \"819a86e9-fe42-4f5b-ae6c-944d2a20a80c\") " pod="llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh"
Oct 30 01:34:41.598373 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:41.598350    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-model-cache\") pod \"e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh\" (UID: \"819a86e9-fe42-4f5b-ae6c-944d2a20a80c\") " pod="llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh"
Oct 30 01:34:41.598431 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:41.598392    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-tmp-dir\") pod \"e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh\" (UID: \"819a86e9-fe42-4f5b-ae6c-944d2a20a80c\") " pod="llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh"
Oct 30 01:34:41.598508 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:41.598488    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-home\") pod \"e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh\" (UID: \"819a86e9-fe42-4f5b-ae6c-944d2a20a80c\") " pod="llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh"
Oct 30 01:34:41.600509 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:41.600484    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-dshm\") pod \"e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh\" (UID: \"819a86e9-fe42-4f5b-ae6c-944d2a20a80c\") " pod="llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh"
Oct 30 01:34:41.600816 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:41.600774    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-tls-certs\") pod \"e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh\" (UID: \"819a86e9-fe42-4f5b-ae6c-944d2a20a80c\") " pod="llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh"
Oct 30 01:34:41.621210 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:41.621181    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7p77\" (UniqueName: \"kubernetes.io/projected/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-kube-api-access-x7p77\") pod \"e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh\" (UID: \"819a86e9-fe42-4f5b-ae6c-944d2a20a80c\") " pod="llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh"
Oct 30 01:34:41.694479 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:41.694383    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh"
Oct 30 01:34:41.997118 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:41.997086    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh"]
Oct 30 01:34:41.998434 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:34:41.998407    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod819a86e9_fe42_4f5b_ae6c_944d2a20a80c.slice/crio-463af41ed2ff069540a89a202312888c3ac52a9e564a8a17468b986a1e0a70d0 WatchSource:0}: Error finding container 463af41ed2ff069540a89a202312888c3ac52a9e564a8a17468b986a1e0a70d0: Status 404 returned error can't find the container with id 463af41ed2ff069540a89a202312888c3ac52a9e564a8a17468b986a1e0a70d0
Oct 30 01:34:42.505258 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:42.505221    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe8356-57d4bpvtqx" podUID="e90e9c5f-6d82-4b96-a4c1-f729e0f33080" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:34:42.505692 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:42.505222    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe8356-79cd4c697vj9c4" podUID="126a90f2-6867-4d83-8b4f-89e1939b492c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 01:34:42.927042 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:42.926994    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh" event={"ID":"819a86e9-fe42-4f5b-ae6c-944d2a20a80c","Type":"ContainerStarted","Data":"f22611c5eb1be361a55ce599737e6b00dfe696d96c57d619c05d5b4cb2311a1d"}
Oct 30 01:34:42.927042 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:42.927048    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh" event={"ID":"819a86e9-fe42-4f5b-ae6c-944d2a20a80c","Type":"ContainerStarted","Data":"463af41ed2ff069540a89a202312888c3ac52a9e564a8a17468b986a1e0a70d0"}
Oct 30 01:34:42.966732 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:42.966680    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh" podStartSLOduration=1.9666675200000001 podStartE2EDuration="1.96666752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:42.965799985 +0000 UTC m=+3773.353476365" watchObservedRunningTime="2026-10-30 01:34:42.96666752 +0000 UTC m=+3773.354343908"
Oct 30 01:34:43.695323 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:43.695289    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh"
Oct 30 01:34:51.350192 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:51.350165    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe8356-79cd4c697vj9c4"
Oct 30 01:34:51.374908 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:51.374886    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe8356-57d4bpvtqx"
Oct 30 01:34:51.523722 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:51.523691    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e90e9c5f-6d82-4b96-a4c1-f729e0f33080-kube-api-access-9q9bg\" (UniqueName: \"kubernetes.io/projected/e90e9c5f-6d82-4b96-a4c1-f729e0f33080-kube-api-access-9q9bg\") pod \"e90e9c5f-6d82-4b96-a4c1-f729e0f33080\" (UID: \"e90e9c5f-6d82-4b96-a4c1-f729e0f33080\") "
Oct 30 01:34:51.523886 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:51.523810    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/126a90f2-6867-4d83-8b4f-89e1939b492c-kube-api-access-42rb9\" (UniqueName: \"kubernetes.io/projected/126a90f2-6867-4d83-8b4f-89e1939b492c-kube-api-access-42rb9\") pod \"126a90f2-6867-4d83-8b4f-89e1939b492c\" (UID: \"126a90f2-6867-4d83-8b4f-89e1939b492c\") "
Oct 30 01:34:51.523946 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:51.523894    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e90e9c5f-6d82-4b96-a4c1-f729e0f33080-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e90e9c5f-6d82-4b96-a4c1-f729e0f33080-plugins-config-volume\") pod \"e90e9c5f-6d82-4b96-a4c1-f729e0f33080\" (UID: \"e90e9c5f-6d82-4b96-a4c1-f729e0f33080\") "
Oct 30 01:34:51.524000 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:51.523983    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/126a90f2-6867-4d83-8b4f-89e1939b492c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/126a90f2-6867-4d83-8b4f-89e1939b492c-plugins-config-volume\") pod \"126a90f2-6867-4d83-8b4f-89e1939b492c\" (UID: \"126a90f2-6867-4d83-8b4f-89e1939b492c\") "
Oct 30 01:34:51.524296 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:51.524270    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e90e9c5f-6d82-4b96-a4c1-f729e0f33080-plugins-config-volume" pod "e90e9c5f-6d82-4b96-a4c1-f729e0f33080" (UID: "e90e9c5f-6d82-4b96-a4c1-f729e0f33080"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:34:51.524385 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:51.524309    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/126a90f2-6867-4d83-8b4f-89e1939b492c-plugins-config-volume" pod "126a90f2-6867-4d83-8b4f-89e1939b492c" (UID: "126a90f2-6867-4d83-8b4f-89e1939b492c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:34:51.524385 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:51.524359    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e90e9c5f-6d82-4b96-a4c1-f729e0f33080-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:34:51.526590 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:51.526573    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e90e9c5f-6d82-4b96-a4c1-f729e0f33080-kube-api-access-9q9bg" pod "e90e9c5f-6d82-4b96-a4c1-f729e0f33080" (UID: "e90e9c5f-6d82-4b96-a4c1-f729e0f33080"). InnerVolumeSpecName "kube-api-access-9q9bg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:34:51.526590 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:51.526584    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/126a90f2-6867-4d83-8b4f-89e1939b492c-kube-api-access-42rb9" pod "126a90f2-6867-4d83-8b4f-89e1939b492c" (UID: "126a90f2-6867-4d83-8b4f-89e1939b492c"). InnerVolumeSpecName "kube-api-access-42rb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:34:51.625253 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:51.625230    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42rb9\" (UniqueName: \"kubernetes.io/projected/126a90f2-6867-4d83-8b4f-89e1939b492c-kube-api-access-42rb9\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:34:51.625253 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:51.625256    4393 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/126a90f2-6867-4d83-8b4f-89e1939b492c-plugins-config-volume\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:34:51.625415 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:51.625265    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q9bg\" (UniqueName: \"kubernetes.io/projected/e90e9c5f-6d82-4b96-a4c1-f729e0f33080-kube-api-access-9q9bg\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:34:51.695496 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:51.695469    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh"
Oct 30 01:34:51.708076 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:51.708052    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh"
Oct 30 01:34:51.998341 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:51.998248    4393 generic.go:356] "Generic (PLEG): container finished" podID="e90e9c5f-6d82-4b96-a4c1-f729e0f33080" containerID="09a2b33392ff16d73b30c961fd93843b64adbc515c5f831d28f4ede0cab2970c" exitCode=0
Oct 30 01:34:51.998341 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:51.998293    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe8356-57d4bpvtqx"
Oct 30 01:34:51.998582 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:51.998332    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe8356-57d4bpvtqx" event={"ID":"e90e9c5f-6d82-4b96-a4c1-f729e0f33080","Type":"ContainerDied","Data":"09a2b33392ff16d73b30c961fd93843b64adbc515c5f831d28f4ede0cab2970c"}
Oct 30 01:34:51.998582 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:51.998391    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe8356-57d4bpvtqx" event={"ID":"e90e9c5f-6d82-4b96-a4c1-f729e0f33080","Type":"ContainerDied","Data":"573a1455cd82207c0b3c74c269bfd379606fe5af553f34f9a1e84b2acef959c5"}
Oct 30 01:34:51.998582 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:51.998421    4393 scope.go:122] "RemoveContainer" containerID="09a2b33392ff16d73b30c961fd93843b64adbc515c5f831d28f4ede0cab2970c"
Oct 30 01:34:51.999852 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:51.999832    4393 generic.go:356] "Generic (PLEG): container finished" podID="126a90f2-6867-4d83-8b4f-89e1939b492c" containerID="a488bb199ac4687841fd951c1fc9f04b0e2653fb50bd8c95c13591d112c2dce1" exitCode=0
Oct 30 01:34:51.999974 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:51.999871    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe8356-79cd4c697vj9c4"
Oct 30 01:34:51.999974 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:51.999893    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe8356-79cd4c697vj9c4" event={"ID":"126a90f2-6867-4d83-8b4f-89e1939b492c","Type":"ContainerDied","Data":"a488bb199ac4687841fd951c1fc9f04b0e2653fb50bd8c95c13591d112c2dce1"}
Oct 30 01:34:51.999974 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:51.999934    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fe8356-79cd4c697vj9c4" event={"ID":"126a90f2-6867-4d83-8b4f-89e1939b492c","Type":"ContainerDied","Data":"ad0936c546d543b0b1b550b7cc086a956bcedacc3d65a2776c62c6653be1d328"}
Oct 30 01:34:52.009836 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:52.009818    4393 scope.go:122] "RemoveContainer" containerID="09a2b33392ff16d73b30c961fd93843b64adbc515c5f831d28f4ede0cab2970c"
Oct 30 01:34:52.010130 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:34:52.010100    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09a2b33392ff16d73b30c961fd93843b64adbc515c5f831d28f4ede0cab2970c\": container with ID starting with 09a2b33392ff16d73b30c961fd93843b64adbc515c5f831d28f4ede0cab2970c not found: ID does not exist" containerID="09a2b33392ff16d73b30c961fd93843b64adbc515c5f831d28f4ede0cab2970c"
Oct 30 01:34:52.010231 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:52.010128    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09a2b33392ff16d73b30c961fd93843b64adbc515c5f831d28f4ede0cab2970c"} err="failed to get container status \"09a2b33392ff16d73b30c961fd93843b64adbc515c5f831d28f4ede0cab2970c\": rpc error: code = NotFound desc = could not find container \"09a2b33392ff16d73b30c961fd93843b64adbc515c5f831d28f4ede0cab2970c\": container with ID starting with 09a2b33392ff16d73b30c961fd93843b64adbc515c5f831d28f4ede0cab2970c not found: ID does not exist"
Oct 30 01:34:52.010231 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:52.010144    4393 scope.go:122] "RemoveContainer" containerID="a488bb199ac4687841fd951c1fc9f04b0e2653fb50bd8c95c13591d112c2dce1"
Oct 30 01:34:52.014144 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:52.014125    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh"
Oct 30 01:34:52.018881 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:52.018762    4393 scope.go:122] "RemoveContainer" containerID="a488bb199ac4687841fd951c1fc9f04b0e2653fb50bd8c95c13591d112c2dce1"
Oct 30 01:34:52.019063 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:34:52.019045    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a488bb199ac4687841fd951c1fc9f04b0e2653fb50bd8c95c13591d112c2dce1\": container with ID starting with a488bb199ac4687841fd951c1fc9f04b0e2653fb50bd8c95c13591d112c2dce1 not found: ID does not exist" containerID="a488bb199ac4687841fd951c1fc9f04b0e2653fb50bd8c95c13591d112c2dce1"
Oct 30 01:34:52.019122 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:52.019071    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a488bb199ac4687841fd951c1fc9f04b0e2653fb50bd8c95c13591d112c2dce1"} err="failed to get container status \"a488bb199ac4687841fd951c1fc9f04b0e2653fb50bd8c95c13591d112c2dce1\": rpc error: code = NotFound desc = could not find container \"a488bb199ac4687841fd951c1fc9f04b0e2653fb50bd8c95c13591d112c2dce1\": container with ID starting with a488bb199ac4687841fd951c1fc9f04b0e2653fb50bd8c95c13591d112c2dce1 not found: ID does not exist"
Oct 30 01:34:52.048242 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:52.048217    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fe8356-79cd4c697vj9c4"]
Oct 30 01:34:52.053730 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:52.053710    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fe8356-79cd4c697vj9c4"]
Oct 30 01:34:52.085673 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:52.085649    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe8356-57d4bpvtqx"]
Oct 30 01:34:52.092126 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:52.092105    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fe8356-57d4bpvtqx"]
Oct 30 01:34:53.730881 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:53.730837    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="126a90f2-6867-4d83-8b4f-89e1939b492c" path="/var/lib/kubelet/pods/126a90f2-6867-4d83-8b4f-89e1939b492c/volumes"
Oct 30 01:34:53.731297 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:53.731275    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e90e9c5f-6d82-4b96-a4c1-f729e0f33080" path="/var/lib/kubelet/pods/e90e9c5f-6d82-4b96-a4c1-f729e0f33080/volumes"
Oct 30 01:34:56.229870 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:56.229828    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv"]
Oct 30 01:34:56.230578 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:56.230552    4393 cpu_manager.go:414] "containerMap: removing container" podUID="e90e9c5f-6d82-4b96-a4c1-f729e0f33080" containerName="payload-pre-processing"
Oct 30 01:34:56.230698 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:56.230581    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e90e9c5f-6d82-4b96-a4c1-f729e0f33080" containerName="payload-pre-processing"
Oct 30 01:34:56.230698 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:56.230649    4393 cpu_manager.go:414] "containerMap: removing container" podUID="126a90f2-6867-4d83-8b4f-89e1939b492c" containerName="payload-processing"
Oct 30 01:34:56.230698 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:56.230660    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="126a90f2-6867-4d83-8b4f-89e1939b492c" containerName="payload-processing"
Oct 30 01:34:56.230860 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:56.230796    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="e90e9c5f-6d82-4b96-a4c1-f729e0f33080" containerName="payload-pre-processing"
Oct 30 01:34:56.230860 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:56.230821    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="126a90f2-6867-4d83-8b4f-89e1939b492c" containerName="payload-processing"
Oct 30 01:34:56.234310 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:56.234293    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv"
Oct 30 01:34:56.260733 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:56.260707    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-e28ec8a1-kserve-self-signed-certs\""
Oct 30 01:34:56.263359 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:56.263337    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv"]
Oct 30 01:34:56.269367 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:56.269343    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc51a33-9bcf-481b-ba6e-bd72049bad22-model-cache\") pod \"e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv\" (UID: \"cdc51a33-9bcf-481b-ba6e-bd72049bad22\") " pod="llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv"
Oct 30 01:34:56.269452 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:56.269401    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc51a33-9bcf-481b-ba6e-bd72049bad22-tls-certs\") pod \"e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv\" (UID: \"cdc51a33-9bcf-481b-ba6e-bd72049bad22\") " pod="llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv"
Oct 30 01:34:56.269518 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:56.269490    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w8m8\" (UniqueName: \"kubernetes.io/projected/cdc51a33-9bcf-481b-ba6e-bd72049bad22-kube-api-access-8w8m8\") pod \"e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv\" (UID: \"cdc51a33-9bcf-481b-ba6e-bd72049bad22\") " pod="llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv"
Oct 30 01:34:56.269614 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:56.269598    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc51a33-9bcf-481b-ba6e-bd72049bad22-dshm\") pod \"e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv\" (UID: \"cdc51a33-9bcf-481b-ba6e-bd72049bad22\") " pod="llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv"
Oct 30 01:34:56.269658 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:56.269631    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc51a33-9bcf-481b-ba6e-bd72049bad22-tmp-dir\") pod \"e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv\" (UID: \"cdc51a33-9bcf-481b-ba6e-bd72049bad22\") " pod="llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv"
Oct 30 01:34:56.269693 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:56.269656    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc51a33-9bcf-481b-ba6e-bd72049bad22-home\") pod \"e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv\" (UID: \"cdc51a33-9bcf-481b-ba6e-bd72049bad22\") " pod="llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv"
Oct 30 01:34:56.370209 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:56.370171    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc51a33-9bcf-481b-ba6e-bd72049bad22-tls-certs\") pod \"e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv\" (UID: \"cdc51a33-9bcf-481b-ba6e-bd72049bad22\") " pod="llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv"
Oct 30 01:34:56.370209 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:56.370212    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w8m8\" (UniqueName: \"kubernetes.io/projected/cdc51a33-9bcf-481b-ba6e-bd72049bad22-kube-api-access-8w8m8\") pod \"e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv\" (UID: \"cdc51a33-9bcf-481b-ba6e-bd72049bad22\") " pod="llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv"
Oct 30 01:34:56.370483 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:56.370259    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc51a33-9bcf-481b-ba6e-bd72049bad22-dshm\") pod \"e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv\" (UID: \"cdc51a33-9bcf-481b-ba6e-bd72049bad22\") " pod="llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv"
Oct 30 01:34:56.370483 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:56.370279    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc51a33-9bcf-481b-ba6e-bd72049bad22-tmp-dir\") pod \"e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv\" (UID: \"cdc51a33-9bcf-481b-ba6e-bd72049bad22\") " pod="llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv"
Oct 30 01:34:56.370483 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:56.370294    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc51a33-9bcf-481b-ba6e-bd72049bad22-home\") pod \"e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv\" (UID: \"cdc51a33-9bcf-481b-ba6e-bd72049bad22\") " pod="llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv"
Oct 30 01:34:56.370483 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:56.370323    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc51a33-9bcf-481b-ba6e-bd72049bad22-model-cache\") pod \"e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv\" (UID: \"cdc51a33-9bcf-481b-ba6e-bd72049bad22\") " pod="llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv"
Oct 30 01:34:56.370800 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:56.370746    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc51a33-9bcf-481b-ba6e-bd72049bad22-tmp-dir\") pod \"e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv\" (UID: \"cdc51a33-9bcf-481b-ba6e-bd72049bad22\") " pod="llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv"
Oct 30 01:34:56.370940 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:56.370847    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc51a33-9bcf-481b-ba6e-bd72049bad22-model-cache\") pod \"e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv\" (UID: \"cdc51a33-9bcf-481b-ba6e-bd72049bad22\") " pod="llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv"
Oct 30 01:34:56.371019 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:56.370955    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc51a33-9bcf-481b-ba6e-bd72049bad22-home\") pod \"e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv\" (UID: \"cdc51a33-9bcf-481b-ba6e-bd72049bad22\") " pod="llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv"
Oct 30 01:34:56.372836 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:56.372811    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc51a33-9bcf-481b-ba6e-bd72049bad22-dshm\") pod \"e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv\" (UID: \"cdc51a33-9bcf-481b-ba6e-bd72049bad22\") " pod="llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv"
Oct 30 01:34:56.373116 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:56.373098    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc51a33-9bcf-481b-ba6e-bd72049bad22-tls-certs\") pod \"e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv\" (UID: \"cdc51a33-9bcf-481b-ba6e-bd72049bad22\") " pod="llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv"
Oct 30 01:34:56.392680 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:56.392658    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w8m8\" (UniqueName: \"kubernetes.io/projected/cdc51a33-9bcf-481b-ba6e-bd72049bad22-kube-api-access-8w8m8\") pod \"e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv\" (UID: \"cdc51a33-9bcf-481b-ba6e-bd72049bad22\") " pod="llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv"
Oct 30 01:34:56.512005 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:34:56.511916    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:34:56.546249 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:56.546208    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv"
Oct 30 01:34:56.847108 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:56.847076    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv"]
Oct 30 01:34:56.848996 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:34:56.848969    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcdc51a33_9bcf_481b_ba6e_bd72049bad22.slice/crio-44d1dc13e195f03d67fda4d67431df990c9a45aca235cf6041d55a3668d09d32 WatchSource:0}: Error finding container 44d1dc13e195f03d67fda4d67431df990c9a45aca235cf6041d55a3668d09d32: Status 404 returned error can't find the container with id 44d1dc13e195f03d67fda4d67431df990c9a45aca235cf6041d55a3668d09d32
Oct 30 01:34:57.029395 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:57.029355    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv" event={"ID":"cdc51a33-9bcf-481b-ba6e-bd72049bad22","Type":"ContainerStarted","Data":"a466c838882efb9ecd9e3d581bec46d50a2d1817c6433f8c39329438ae544d66"}
Oct 30 01:34:57.029395 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:57.029402    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv" event={"ID":"cdc51a33-9bcf-481b-ba6e-bd72049bad22","Type":"ContainerStarted","Data":"44d1dc13e195f03d67fda4d67431df990c9a45aca235cf6041d55a3668d09d32"}
Oct 30 01:34:57.070346 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:57.070295    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv" podStartSLOduration=1.07027846 podStartE2EDuration="1.07027846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:57.068551522 +0000 UTC m=+3787.456227910" watchObservedRunningTime="2026-10-30 01:34:57.07027846 +0000 UTC m=+3787.457954849"
Oct 30 01:34:57.547259 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:34:57.547222    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv"
Oct 30 01:35:02.284967 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:02.284917    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-97m56 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:35:02.287474 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:02.285034    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-97m56" podUID="d4da6381-6d65-4f27-9f7f-907dd8b5da23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:35:06.546390 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:06.546355    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv"
Oct 30 01:35:06.558670 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:06.558644    4393 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv"
Oct 30 01:35:07.123134 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:07.123106    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv"
Oct 30 01:35:26.510355 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:35:26.510312    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:35:32.096736 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:32.096637    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh"]
Oct 30 01:35:37.300962 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:37.300925    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-cngvb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:35:37.301477 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:37.300991    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-cngvb" podUID="f3bd5875-a930-4985-b820-efe272d30763" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:35:37.752473 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:37.752431    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv"]
Oct 30 01:35:47.119209 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:47.119162    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh" podUID="819a86e9-fe42-4f5b-ae6c-944d2a20a80c" containerName="main" containerID="cri-o://f22611c5eb1be361a55ce599737e6b00dfe696d96c57d619c05d5b4cb2311a1d" gracePeriod=45
Oct 30 01:35:47.375705 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:47.375629    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh"
Oct 30 01:35:47.389080 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:47.389045    4393 generic.go:356] "Generic (PLEG): container finished" podID="819a86e9-fe42-4f5b-ae6c-944d2a20a80c" containerID="f22611c5eb1be361a55ce599737e6b00dfe696d96c57d619c05d5b4cb2311a1d" exitCode=0
Oct 30 01:35:47.389206 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:47.389085    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh"
Oct 30 01:35:47.389206 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:47.389123    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh" event={"ID":"819a86e9-fe42-4f5b-ae6c-944d2a20a80c","Type":"ContainerDied","Data":"f22611c5eb1be361a55ce599737e6b00dfe696d96c57d619c05d5b4cb2311a1d"}
Oct 30 01:35:47.389206 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:47.389174    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh" event={"ID":"819a86e9-fe42-4f5b-ae6c-944d2a20a80c","Type":"ContainerDied","Data":"463af41ed2ff069540a89a202312888c3ac52a9e564a8a17468b986a1e0a70d0"}
Oct 30 01:35:47.389206 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:47.389194    4393 scope.go:122] "RemoveContainer" containerID="f22611c5eb1be361a55ce599737e6b00dfe696d96c57d619c05d5b4cb2311a1d"
Oct 30 01:35:47.398262 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:47.398245    4393 scope.go:122] "RemoveContainer" containerID="f22611c5eb1be361a55ce599737e6b00dfe696d96c57d619c05d5b4cb2311a1d"
Oct 30 01:35:47.398547 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:35:47.398526    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f22611c5eb1be361a55ce599737e6b00dfe696d96c57d619c05d5b4cb2311a1d\": container with ID starting with f22611c5eb1be361a55ce599737e6b00dfe696d96c57d619c05d5b4cb2311a1d not found: ID does not exist" containerID="f22611c5eb1be361a55ce599737e6b00dfe696d96c57d619c05d5b4cb2311a1d"
Oct 30 01:35:47.398624 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:47.398560    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f22611c5eb1be361a55ce599737e6b00dfe696d96c57d619c05d5b4cb2311a1d"} err="failed to get container status \"f22611c5eb1be361a55ce599737e6b00dfe696d96c57d619c05d5b4cb2311a1d\": rpc error: code = NotFound desc = could not find container \"f22611c5eb1be361a55ce599737e6b00dfe696d96c57d619c05d5b4cb2311a1d\": container with ID starting with f22611c5eb1be361a55ce599737e6b00dfe696d96c57d619c05d5b4cb2311a1d not found: ID does not exist"
Oct 30 01:35:47.434193 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:47.434163    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-tls-certs\" (UniqueName: \"kubernetes.io/secret/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-tls-certs\") pod \"819a86e9-fe42-4f5b-ae6c-944d2a20a80c\" (UID: \"819a86e9-fe42-4f5b-ae6c-944d2a20a80c\") "
Oct 30 01:35:47.434361 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:47.434234    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-home\" (UniqueName: \"kubernetes.io/empty-dir/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-home\") pod \"819a86e9-fe42-4f5b-ae6c-944d2a20a80c\" (UID: \"819a86e9-fe42-4f5b-ae6c-944d2a20a80c\") "
Oct 30 01:35:47.434361 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:47.434286    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-model-cache\") pod \"819a86e9-fe42-4f5b-ae6c-944d2a20a80c\" (UID: \"819a86e9-fe42-4f5b-ae6c-944d2a20a80c\") "
Oct 30 01:35:47.434361 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:47.434320    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-kube-api-access-x7p77\" (UniqueName: \"kubernetes.io/projected/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-kube-api-access-x7p77\") pod \"819a86e9-fe42-4f5b-ae6c-944d2a20a80c\" (UID: \"819a86e9-fe42-4f5b-ae6c-944d2a20a80c\") "
Oct 30 01:35:47.434361 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:47.434342    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-tmp-dir\") pod \"819a86e9-fe42-4f5b-ae6c-944d2a20a80c\" (UID: \"819a86e9-fe42-4f5b-ae6c-944d2a20a80c\") "
Oct 30 01:35:47.434608 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:47.434429    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-dshm\") pod \"819a86e9-fe42-4f5b-ae6c-944d2a20a80c\" (UID: \"819a86e9-fe42-4f5b-ae6c-944d2a20a80c\") "
Oct 30 01:35:47.434608 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:47.434497    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-home" pod "819a86e9-fe42-4f5b-ae6c-944d2a20a80c" (UID: "819a86e9-fe42-4f5b-ae6c-944d2a20a80c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:35:47.434729 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:47.434603    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-model-cache" pod "819a86e9-fe42-4f5b-ae6c-944d2a20a80c" (UID: "819a86e9-fe42-4f5b-ae6c-944d2a20a80c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:35:47.434808 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:47.434743    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-tmp-dir" pod "819a86e9-fe42-4f5b-ae6c-944d2a20a80c" (UID: "819a86e9-fe42-4f5b-ae6c-944d2a20a80c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:35:47.434870 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:47.434857    4393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-home\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:35:47.434941 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:47.434877    4393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-model-cache\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:35:47.434941 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:47.434889    4393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-tmp-dir\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:35:47.436863 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:47.436830    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-tls-certs" pod "819a86e9-fe42-4f5b-ae6c-944d2a20a80c" (UID: "819a86e9-fe42-4f5b-ae6c-944d2a20a80c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:35:47.436863 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:47.436844    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-dshm" pod "819a86e9-fe42-4f5b-ae6c-944d2a20a80c" (UID: "819a86e9-fe42-4f5b-ae6c-944d2a20a80c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:35:47.437107 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:47.437082    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-kube-api-access-x7p77" pod "819a86e9-fe42-4f5b-ae6c-944d2a20a80c" (UID: "819a86e9-fe42-4f5b-ae6c-944d2a20a80c"). InnerVolumeSpecName "kube-api-access-x7p77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:35:47.535868 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:47.535836    4393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-tls-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:35:47.535868 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:47.535868    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7p77\" (UniqueName: \"kubernetes.io/projected/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-kube-api-access-x7p77\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:35:47.536052 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:47.535881    4393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/819a86e9-fe42-4f5b-ae6c-944d2a20a80c-dshm\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:35:47.733516 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:47.733489    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh"]
Oct 30 01:35:47.745227 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:47.745195    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-e28ec8a1-kserve-5bc585cbd5-nrqdh"]
Oct 30 01:35:49.724053 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:49.724017    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="819a86e9-fe42-4f5b-ae6c-944d2a20a80c" path="/var/lib/kubelet/pods/819a86e9-fe42-4f5b-ae6c-944d2a20a80c/volumes"
Oct 30 01:35:52.781480 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:52.781435    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv" podUID="cdc51a33-9bcf-481b-ba6e-bd72049bad22" containerName="main" containerID="cri-o://a466c838882efb9ecd9e3d581bec46d50a2d1817c6433f8c39329438ae544d66" gracePeriod=45
Oct 30 01:35:53.040060 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:53.039996    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv"
Oct 30 01:35:53.091220 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:53.091189    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc51a33-9bcf-481b-ba6e-bd72049bad22-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc51a33-9bcf-481b-ba6e-bd72049bad22-tmp-dir\") pod \"cdc51a33-9bcf-481b-ba6e-bd72049bad22\" (UID: \"cdc51a33-9bcf-481b-ba6e-bd72049bad22\") "
Oct 30 01:35:53.091405 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:53.091301    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc51a33-9bcf-481b-ba6e-bd72049bad22-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc51a33-9bcf-481b-ba6e-bd72049bad22-model-cache\") pod \"cdc51a33-9bcf-481b-ba6e-bd72049bad22\" (UID: \"cdc51a33-9bcf-481b-ba6e-bd72049bad22\") "
Oct 30 01:35:53.091405 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:53.091337    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdc51a33-9bcf-481b-ba6e-bd72049bad22-kube-api-access-8w8m8\" (UniqueName: \"kubernetes.io/projected/cdc51a33-9bcf-481b-ba6e-bd72049bad22-kube-api-access-8w8m8\") pod \"cdc51a33-9bcf-481b-ba6e-bd72049bad22\" (UID: \"cdc51a33-9bcf-481b-ba6e-bd72049bad22\") "
Oct 30 01:35:53.091405 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:53.091365    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc51a33-9bcf-481b-ba6e-bd72049bad22-home\" (UniqueName: \"kubernetes.io/empty-dir/cdc51a33-9bcf-481b-ba6e-bd72049bad22-home\") pod \"cdc51a33-9bcf-481b-ba6e-bd72049bad22\" (UID: \"cdc51a33-9bcf-481b-ba6e-bd72049bad22\") "
Oct 30 01:35:53.091743 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:53.091460    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc51a33-9bcf-481b-ba6e-bd72049bad22-tmp-dir" pod "cdc51a33-9bcf-481b-ba6e-bd72049bad22" (UID: "cdc51a33-9bcf-481b-ba6e-bd72049bad22"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:35:53.091743 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:53.091581    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc51a33-9bcf-481b-ba6e-bd72049bad22-model-cache" pod "cdc51a33-9bcf-481b-ba6e-bd72049bad22" (UID: "cdc51a33-9bcf-481b-ba6e-bd72049bad22"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:35:53.091743 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:53.091593    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdc51a33-9bcf-481b-ba6e-bd72049bad22-tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc51a33-9bcf-481b-ba6e-bd72049bad22-tls-certs\") pod \"cdc51a33-9bcf-481b-ba6e-bd72049bad22\" (UID: \"cdc51a33-9bcf-481b-ba6e-bd72049bad22\") "
Oct 30 01:35:53.091743 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:53.091666    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc51a33-9bcf-481b-ba6e-bd72049bad22-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc51a33-9bcf-481b-ba6e-bd72049bad22-dshm\") pod \"cdc51a33-9bcf-481b-ba6e-bd72049bad22\" (UID: \"cdc51a33-9bcf-481b-ba6e-bd72049bad22\") "
Oct 30 01:35:53.092107 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:53.091843    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc51a33-9bcf-481b-ba6e-bd72049bad22-home" pod "cdc51a33-9bcf-481b-ba6e-bd72049bad22" (UID: "cdc51a33-9bcf-481b-ba6e-bd72049bad22"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:35:53.092173 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:53.092154    4393 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc51a33-9bcf-481b-ba6e-bd72049bad22-tmp-dir\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:35:53.092395 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:53.092176    4393 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc51a33-9bcf-481b-ba6e-bd72049bad22-model-cache\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:35:53.092395 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:53.092189    4393 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc51a33-9bcf-481b-ba6e-bd72049bad22-home\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:35:53.094129 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:53.094102    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc51a33-9bcf-481b-ba6e-bd72049bad22-kube-api-access-8w8m8" pod "cdc51a33-9bcf-481b-ba6e-bd72049bad22" (UID: "cdc51a33-9bcf-481b-ba6e-bd72049bad22"). InnerVolumeSpecName "kube-api-access-8w8m8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:35:53.094296 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:53.094274    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc51a33-9bcf-481b-ba6e-bd72049bad22-dshm" pod "cdc51a33-9bcf-481b-ba6e-bd72049bad22" (UID: "cdc51a33-9bcf-481b-ba6e-bd72049bad22"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 01:35:53.094518 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:53.094489    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdc51a33-9bcf-481b-ba6e-bd72049bad22-tls-certs" pod "cdc51a33-9bcf-481b-ba6e-bd72049bad22" (UID: "cdc51a33-9bcf-481b-ba6e-bd72049bad22"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:35:53.196805 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:53.196443    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w8m8\" (UniqueName: \"kubernetes.io/projected/cdc51a33-9bcf-481b-ba6e-bd72049bad22-kube-api-access-8w8m8\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:35:53.196805 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:53.196496    4393 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc51a33-9bcf-481b-ba6e-bd72049bad22-tls-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:35:53.196805 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:53.196513    4393 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc51a33-9bcf-481b-ba6e-bd72049bad22-dshm\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:35:53.454814 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:53.446952    4393 generic.go:356] "Generic (PLEG): container finished" podID="cdc51a33-9bcf-481b-ba6e-bd72049bad22" containerID="a466c838882efb9ecd9e3d581bec46d50a2d1817c6433f8c39329438ae544d66" exitCode=0
Oct 30 01:35:53.454814 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:53.447089    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv" event={"ID":"cdc51a33-9bcf-481b-ba6e-bd72049bad22","Type":"ContainerDied","Data":"a466c838882efb9ecd9e3d581bec46d50a2d1817c6433f8c39329438ae544d66"}
Oct 30 01:35:53.454814 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:53.447127    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv" event={"ID":"cdc51a33-9bcf-481b-ba6e-bd72049bad22","Type":"ContainerDied","Data":"44d1dc13e195f03d67fda4d67431df990c9a45aca235cf6041d55a3668d09d32"}
Oct 30 01:35:53.454814 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:53.447150    4393 scope.go:122] "RemoveContainer" containerID="a466c838882efb9ecd9e3d581bec46d50a2d1817c6433f8c39329438ae544d66"
Oct 30 01:35:53.454814 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:53.447300    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv"
Oct 30 01:35:53.501485 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:53.501467    4393 scope.go:122] "RemoveContainer" containerID="a466c838882efb9ecd9e3d581bec46d50a2d1817c6433f8c39329438ae544d66"
Oct 30 01:35:53.507472 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:35:53.501803    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a466c838882efb9ecd9e3d581bec46d50a2d1817c6433f8c39329438ae544d66\": container with ID starting with a466c838882efb9ecd9e3d581bec46d50a2d1817c6433f8c39329438ae544d66 not found: ID does not exist" containerID="a466c838882efb9ecd9e3d581bec46d50a2d1817c6433f8c39329438ae544d66"
Oct 30 01:35:53.507472 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:53.501849    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a466c838882efb9ecd9e3d581bec46d50a2d1817c6433f8c39329438ae544d66"} err="failed to get container status \"a466c838882efb9ecd9e3d581bec46d50a2d1817c6433f8c39329438ae544d66\": rpc error: code = NotFound desc = could not find container \"a466c838882efb9ecd9e3d581bec46d50a2d1817c6433f8c39329438ae544d66\": container with ID starting with a466c838882efb9ecd9e3d581bec46d50a2d1817c6433f8c39329438ae544d66 not found: ID does not exist"
Oct 30 01:35:53.565900 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:53.565858    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv"]
Oct 30 01:35:53.569763 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:53.569739    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-e28ec8a1-kserve-5846d6f465-vn5mv"]
Oct 30 01:35:53.769850 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:35:53.769215    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdc51a33-9bcf-481b-ba6e-bd72049bad22" path="/var/lib/kubelet/pods/cdc51a33-9bcf-481b-ba6e-bd72049bad22/volumes"
Oct 30 01:35:56.519813 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:35:56.519143    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:36:17.434524 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:17.434487    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-62jgl"]
Oct 30 01:36:17.435069 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:17.435051    4393 cpu_manager.go:414] "containerMap: removing container" podUID="cdc51a33-9bcf-481b-ba6e-bd72049bad22" containerName="main"
Oct 30 01:36:17.435123 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:17.435073    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdc51a33-9bcf-481b-ba6e-bd72049bad22" containerName="main"
Oct 30 01:36:17.435123 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:17.435100    4393 cpu_manager.go:414] "containerMap: removing container" podUID="819a86e9-fe42-4f5b-ae6c-944d2a20a80c" containerName="main"
Oct 30 01:36:17.435123 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:17.435105    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="819a86e9-fe42-4f5b-ae6c-944d2a20a80c" containerName="main"
Oct 30 01:36:17.435229 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:17.435193    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="819a86e9-fe42-4f5b-ae6c-944d2a20a80c" containerName="main"
Oct 30 01:36:17.435229 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:17.435209    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdc51a33-9bcf-481b-ba6e-bd72049bad22" containerName="main"
Oct 30 01:36:17.438500 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:17.438480    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-62jgl"
Oct 30 01:36:17.470942 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:17.470907    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-62jgl"]
Oct 30 01:36:17.568959 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:17.568918    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/763034f0-a047-4401-ae4e-fc5bc807d81a-metrics-certs\") pod \"maas-api-595cc648c-62jgl\" (UID: \"763034f0-a047-4401-ae4e-fc5bc807d81a\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-62jgl"
Oct 30 01:36:17.569153 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:17.568965    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8mtz\" (UniqueName: \"kubernetes.io/projected/763034f0-a047-4401-ae4e-fc5bc807d81a-kube-api-access-s8mtz\") pod \"maas-api-595cc648c-62jgl\" (UID: \"763034f0-a047-4401-ae4e-fc5bc807d81a\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-62jgl"
Oct 30 01:36:17.569153 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:17.569111    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/763034f0-a047-4401-ae4e-fc5bc807d81a-maas-api-tls\") pod \"maas-api-595cc648c-62jgl\" (UID: \"763034f0-a047-4401-ae4e-fc5bc807d81a\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-62jgl"
Oct 30 01:36:17.569153 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:17.569139    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/763034f0-a047-4401-ae4e-fc5bc807d81a-openshift-service-ca\") pod \"maas-api-595cc648c-62jgl\" (UID: \"763034f0-a047-4401-ae4e-fc5bc807d81a\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-62jgl"
Oct 30 01:36:17.670182 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:17.670147    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/763034f0-a047-4401-ae4e-fc5bc807d81a-maas-api-tls\") pod \"maas-api-595cc648c-62jgl\" (UID: \"763034f0-a047-4401-ae4e-fc5bc807d81a\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-62jgl"
Oct 30 01:36:17.670404 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:17.670201    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/763034f0-a047-4401-ae4e-fc5bc807d81a-openshift-service-ca\") pod \"maas-api-595cc648c-62jgl\" (UID: \"763034f0-a047-4401-ae4e-fc5bc807d81a\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-62jgl"
Oct 30 01:36:17.670404 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:17.670278    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/763034f0-a047-4401-ae4e-fc5bc807d81a-metrics-certs\") pod \"maas-api-595cc648c-62jgl\" (UID: \"763034f0-a047-4401-ae4e-fc5bc807d81a\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-62jgl"
Oct 30 01:36:17.670404 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:17.670310    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8mtz\" (UniqueName: \"kubernetes.io/projected/763034f0-a047-4401-ae4e-fc5bc807d81a-kube-api-access-s8mtz\") pod \"maas-api-595cc648c-62jgl\" (UID: \"763034f0-a047-4401-ae4e-fc5bc807d81a\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-62jgl"
Oct 30 01:36:17.670926 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:17.670896    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/763034f0-a047-4401-ae4e-fc5bc807d81a-openshift-service-ca\") pod \"maas-api-595cc648c-62jgl\" (UID: \"763034f0-a047-4401-ae4e-fc5bc807d81a\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-62jgl"
Oct 30 01:36:17.672843 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:17.672813    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/763034f0-a047-4401-ae4e-fc5bc807d81a-metrics-certs\") pod \"maas-api-595cc648c-62jgl\" (UID: \"763034f0-a047-4401-ae4e-fc5bc807d81a\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-62jgl"
Oct 30 01:36:17.672942 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:17.672855    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/763034f0-a047-4401-ae4e-fc5bc807d81a-maas-api-tls\") pod \"maas-api-595cc648c-62jgl\" (UID: \"763034f0-a047-4401-ae4e-fc5bc807d81a\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-62jgl"
Oct 30 01:36:17.693507 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:17.693434    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8mtz\" (UniqueName: \"kubernetes.io/projected/763034f0-a047-4401-ae4e-fc5bc807d81a-kube-api-access-s8mtz\") pod \"maas-api-595cc648c-62jgl\" (UID: \"763034f0-a047-4401-ae4e-fc5bc807d81a\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-62jgl"
Oct 30 01:36:17.754807 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:17.754763    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-62jgl"
Oct 30 01:36:18.060817 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:18.060767    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-62jgl"]
Oct 30 01:36:18.061501 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:36:18.061468    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod763034f0_a047_4401_ae4e_fc5bc807d81a.slice/crio-72f5b6fb15690cb4979081a9fcc013d6f74e14dbdc9662653b6b2f1654e8f96d WatchSource:0}: Error finding container 72f5b6fb15690cb4979081a9fcc013d6f74e14dbdc9662653b6b2f1654e8f96d: Status 404 returned error can't find the container with id 72f5b6fb15690cb4979081a9fcc013d6f74e14dbdc9662653b6b2f1654e8f96d
Oct 30 01:36:18.063002 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:18.062982    4393 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 01:36:18.615411 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:18.615373    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-62jgl" event={"ID":"763034f0-a047-4401-ae4e-fc5bc807d81a","Type":"ContainerStarted","Data":"deef401e0649a2b3fa13e4ae11f6c7a1117622a14242f1e6ca0409cd43c7c84d"}
Oct 30 01:36:18.615984 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:18.615471    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-62jgl"
Oct 30 01:36:18.615984 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:18.615497    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-62jgl" event={"ID":"763034f0-a047-4401-ae4e-fc5bc807d81a","Type":"ContainerStarted","Data":"72f5b6fb15690cb4979081a9fcc013d6f74e14dbdc9662653b6b2f1654e8f96d"}
Oct 30 01:36:18.657897 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:18.657846    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-62jgl" podStartSLOduration=1.263135586 podStartE2EDuration="1.65783011s" totalImagesPullingTime="394.694524ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:36:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:36:18.655282086 +0000 UTC m=+3869.042958476" watchObservedRunningTime="2026-10-30 01:36:18.65783011 +0000 UTC m=+3869.045506499"
Oct 30 01:36:24.627694 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:24.627664    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-62jgl"
Oct 30 01:36:26.520822 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:36:26.520753    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:36:32.108035 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:32.107999    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-27r2g"]
Oct 30 01:36:32.109063 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:32.108291    4393 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-27r2g" podUID="29c80ca8-5215-41d3-9c4c-a4744be8b019" containerName="maas-api" containerID="cri-o://bab2060691968659b00faac6a74e51499b7b94d65c56523d9619dd4f02273f4b" gracePeriod=30
Oct 30 01:36:33.373300 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:33.373274    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-27r2g"
Oct 30 01:36:33.452078 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:33.451996    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29c80ca8-5215-41d3-9c4c-a4744be8b019-kube-api-access-6mkdt\" (UniqueName: \"kubernetes.io/projected/29c80ca8-5215-41d3-9c4c-a4744be8b019-kube-api-access-6mkdt\") pod \"29c80ca8-5215-41d3-9c4c-a4744be8b019\" (UID: \"29c80ca8-5215-41d3-9c4c-a4744be8b019\") "
Oct 30 01:36:33.452256 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:33.452059    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29c80ca8-5215-41d3-9c4c-a4744be8b019-metrics-certs\" (UniqueName: \"kubernetes.io/secret/29c80ca8-5215-41d3-9c4c-a4744be8b019-metrics-certs\") pod \"29c80ca8-5215-41d3-9c4c-a4744be8b019\" (UID: \"29c80ca8-5215-41d3-9c4c-a4744be8b019\") "
Oct 30 01:36:33.452256 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:33.452149    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29c80ca8-5215-41d3-9c4c-a4744be8b019-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29c80ca8-5215-41d3-9c4c-a4744be8b019-maas-api-tls\") pod \"29c80ca8-5215-41d3-9c4c-a4744be8b019\" (UID: \"29c80ca8-5215-41d3-9c4c-a4744be8b019\") "
Oct 30 01:36:33.452256 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:33.452181    4393 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29c80ca8-5215-41d3-9c4c-a4744be8b019-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c80ca8-5215-41d3-9c4c-a4744be8b019-openshift-service-ca\") pod \"29c80ca8-5215-41d3-9c4c-a4744be8b019\" (UID: \"29c80ca8-5215-41d3-9c4c-a4744be8b019\") "
Oct 30 01:36:33.452591 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:33.452564    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29c80ca8-5215-41d3-9c4c-a4744be8b019-openshift-service-ca" pod "29c80ca8-5215-41d3-9c4c-a4744be8b019" (UID: "29c80ca8-5215-41d3-9c4c-a4744be8b019"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 01:36:33.454291 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:33.454267    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29c80ca8-5215-41d3-9c4c-a4744be8b019-maas-api-tls" pod "29c80ca8-5215-41d3-9c4c-a4744be8b019" (UID: "29c80ca8-5215-41d3-9c4c-a4744be8b019"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:36:33.454375 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:33.454289    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29c80ca8-5215-41d3-9c4c-a4744be8b019-metrics-certs" pod "29c80ca8-5215-41d3-9c4c-a4744be8b019" (UID: "29c80ca8-5215-41d3-9c4c-a4744be8b019"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 01:36:33.454425 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:33.454383    4393 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c80ca8-5215-41d3-9c4c-a4744be8b019-kube-api-access-6mkdt" pod "29c80ca8-5215-41d3-9c4c-a4744be8b019" (UID: "29c80ca8-5215-41d3-9c4c-a4744be8b019"). InnerVolumeSpecName "kube-api-access-6mkdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 01:36:33.553983 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:33.553953    4393 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29c80ca8-5215-41d3-9c4c-a4744be8b019-metrics-certs\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:36:33.553983 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:33.553981    4393 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29c80ca8-5215-41d3-9c4c-a4744be8b019-maas-api-tls\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:36:33.553983 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:33.553990    4393 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c80ca8-5215-41d3-9c4c-a4744be8b019-openshift-service-ca\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:36:33.554225 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:33.553999    4393 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mkdt\" (UniqueName: \"kubernetes.io/projected/29c80ca8-5215-41d3-9c4c-a4744be8b019-kube-api-access-6mkdt\") on node \"ip-10-0-135-56.ec2.internal\" DevicePath \"\""
Oct 30 01:36:33.718769 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:33.718687    4393 generic.go:356] "Generic (PLEG): container finished" podID="29c80ca8-5215-41d3-9c4c-a4744be8b019" containerID="bab2060691968659b00faac6a74e51499b7b94d65c56523d9619dd4f02273f4b" exitCode=0
Oct 30 01:36:33.718937 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:33.718800    4393 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-27r2g"
Oct 30 01:36:33.724547 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:33.724519    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-27r2g" event={"ID":"29c80ca8-5215-41d3-9c4c-a4744be8b019","Type":"ContainerDied","Data":"bab2060691968659b00faac6a74e51499b7b94d65c56523d9619dd4f02273f4b"}
Oct 30 01:36:33.724683 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:33.724569    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-27r2g" event={"ID":"29c80ca8-5215-41d3-9c4c-a4744be8b019","Type":"ContainerDied","Data":"2467ffdc17caa4db687d2f210b6f8adc36161beef75a42c2292042f06f034365"}
Oct 30 01:36:33.724683 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:33.724600    4393 scope.go:122] "RemoveContainer" containerID="bab2060691968659b00faac6a74e51499b7b94d65c56523d9619dd4f02273f4b"
Oct 30 01:36:33.735509 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:33.735483    4393 scope.go:122] "RemoveContainer" containerID="bab2060691968659b00faac6a74e51499b7b94d65c56523d9619dd4f02273f4b"
Oct 30 01:36:33.735860 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:36:33.735829    4393 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bab2060691968659b00faac6a74e51499b7b94d65c56523d9619dd4f02273f4b\": container with ID starting with bab2060691968659b00faac6a74e51499b7b94d65c56523d9619dd4f02273f4b not found: ID does not exist" containerID="bab2060691968659b00faac6a74e51499b7b94d65c56523d9619dd4f02273f4b"
Oct 30 01:36:33.735962 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:33.735871    4393 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bab2060691968659b00faac6a74e51499b7b94d65c56523d9619dd4f02273f4b"} err="failed to get container status \"bab2060691968659b00faac6a74e51499b7b94d65c56523d9619dd4f02273f4b\": rpc error: code = NotFound desc = could not find container \"bab2060691968659b00faac6a74e51499b7b94d65c56523d9619dd4f02273f4b\": container with ID starting with bab2060691968659b00faac6a74e51499b7b94d65c56523d9619dd4f02273f4b not found: ID does not exist"
Oct 30 01:36:33.770091 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:33.770065    4393 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-27r2g"]
Oct 30 01:36:33.773823 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:33.773801    4393 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-27r2g"]
Oct 30 01:36:35.754812 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:36:35.754737    4393 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29c80ca8-5215-41d3-9c4c-a4744be8b019" path="/var/lib/kubelet/pods/29c80ca8-5215-41d3-9c4c-a4744be8b019/volumes"
Oct 30 01:36:56.509624 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:36:56.509583    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:37:15.024057 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:15.024003    4393 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7d9897b54c-26zqk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 01:37:15.025771 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:15.025708    4393 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7d9897b54c-26zqk" podUID="58e0c954-cdcc-4ad1-bbf7-a2197816e842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 01:37:23.684397 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:23.684362    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7fd968db48-lmzwp_afe02d9e-6306-48f3-adde-1e5aa9381b83/manager/0.log"
Oct 30 01:37:24.699026 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:24.698996    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6b5b68ff8f-jrgsq_a06f7a71-cfbc-4a87-a668-bb3fa21ee2ef/manager/0.log"
Oct 30 01:37:25.975447 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:25.975417    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bcdb85766-mfpvd_bd326a32-6e36-4753-88fd-2855e415cc15/copy-manifests/0.log"
Oct 30 01:37:26.020162 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:26.020128    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bcdb85766-mfpvd_bd326a32-6e36-4753-88fd-2855e415cc15/manager/0.log"
Oct 30 01:37:26.237404 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:26.237319    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-fn7wx_55f3efa3-7ddf-47e5-be8c-1acb8f1b6d3f/proxy/0.log"
Oct 30 01:37:26.510205 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:37:26.510115    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:37:27.008281 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:27.008249    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-62jgl_763034f0-a047-4401-ae4e-fc5bc807d81a/maas-api/0.log"
Oct 30 01:37:29.056526 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:29.056493    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-qzr7h_42437b4a-aa45-4add-b6ad-e774be593b64/postgres/0.log"
Oct 30 01:37:30.797280 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:30.797250    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-kvgjz_7bd77191-8d97-498a-978b-0765096177a1/manager/0.log"
Oct 30 01:37:31.000490 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:31.000458    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-xgjkr_4c9fdd3e-ff82-404c-a637-1b18a139938d/manager/0.log"
Oct 30 01:37:31.398994 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:31.398963    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-4zr2r_c2d79a11-d151-4368-9469-2034d93cd39d/manager/0.log"
Oct 30 01:37:32.567683 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:32.567643    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw_d685e131-7f49-4c0d-870f-1d1a51bd1b74/istio-proxy/0.log"
Oct 30 01:37:32.957759 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:32.957719    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b7df967f4-5hvrq_e1c4b01c-5605-474b-8e98-91f81fe6873d/kube-auth-proxy/0.log"
Oct 30 01:37:33.341580 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:33.341497    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-26zqk_58e0c954-cdcc-4ad1-bbf7-a2197816e842/istio-proxy/0.log"
Oct 30 01:37:33.747681 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:33.747647    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-97m56_d4da6381-6d65-4f27-9f7f-907dd8b5da23/istio-proxy/0.log"
Oct 30 01:37:34.128334 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:34.128303    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-cngvb_f3bd5875-a930-4985-b820-efe272d30763/istio-proxy/1.log"
Oct 30 01:37:35.475415 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:35.475378    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-l4gst_9217cef0-6bf2-41d5-b303-4c4b78dcb7f0/payload-processing/0.log"
Oct 30 01:37:36.107792 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:36.107752    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-cngvb_f3bd5875-a930-4985-b820-efe272d30763/istio-proxy/0.log"
Oct 30 01:37:36.657749 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:36.657718    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c_3ff1c407-8b92-4657-82ac-eb65af7d9bbe/main/0.log"
Oct 30 01:37:37.238184 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:37.238146    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-s7t4h_770ca287-b172-4664-9e09-f024674f5c9c/main/0.log"
Oct 30 01:37:37.598328 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:37.598256    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2_68d0836f-44b4-45d7-a37f-bd20ce93a93f/main/0.log"
Oct 30 01:37:47.985156 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:47.985117    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7fd968db48-lmzwp_afe02d9e-6306-48f3-adde-1e5aa9381b83/manager/0.log"
Oct 30 01:37:48.923512 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:48.923478    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6b5b68ff8f-jrgsq_a06f7a71-cfbc-4a87-a668-bb3fa21ee2ef/manager/0.log"
Oct 30 01:37:50.390560 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:50.390536    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bcdb85766-mfpvd_bd326a32-6e36-4753-88fd-2855e415cc15/copy-manifests/0.log"
Oct 30 01:37:50.434359 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:50.434322    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bcdb85766-mfpvd_bd326a32-6e36-4753-88fd-2855e415cc15/manager/0.log"
Oct 30 01:37:50.679268 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:50.679175    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-fn7wx_55f3efa3-7ddf-47e5-be8c-1acb8f1b6d3f/proxy/0.log"
Oct 30 01:37:51.246215 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:51.246179    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-62jgl_763034f0-a047-4401-ae4e-fc5bc807d81a/maas-api/0.log"
Oct 30 01:37:53.449921 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:53.449889    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-qzr7h_42437b4a-aa45-4add-b6ad-e774be593b64/postgres/0.log"
Oct 30 01:37:54.886543 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:54.886513    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-kvgjz_7bd77191-8d97-498a-978b-0765096177a1/manager/0.log"
Oct 30 01:37:55.119899 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:55.119868    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-xgjkr_4c9fdd3e-ff82-404c-a637-1b18a139938d/manager/0.log"
Oct 30 01:37:55.518586 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:55.518557    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-4zr2r_c2d79a11-d151-4368-9469-2034d93cd39d/manager/0.log"
Oct 30 01:37:56.510287 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:37:56.510250    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:37:56.558102 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:56.558061    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw_d685e131-7f49-4c0d-870f-1d1a51bd1b74/istio-proxy/0.log"
Oct 30 01:37:56.951428 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:56.951395    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b7df967f4-5hvrq_e1c4b01c-5605-474b-8e98-91f81fe6873d/kube-auth-proxy/0.log"
Oct 30 01:37:57.331744 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:57.331631    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-26zqk_58e0c954-cdcc-4ad1-bbf7-a2197816e842/istio-proxy/0.log"
Oct 30 01:37:57.724634 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:57.724609    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-97m56_d4da6381-6d65-4f27-9f7f-907dd8b5da23/istio-proxy/0.log"
Oct 30 01:37:58.113343 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:58.113259    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-cngvb_f3bd5875-a930-4985-b820-efe272d30763/istio-proxy/1.log"
Oct 30 01:37:59.474303 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:37:59.474268    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-l4gst_9217cef0-6bf2-41d5-b303-4c4b78dcb7f0/payload-processing/0.log"
Oct 30 01:38:00.251339 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:38:00.251306    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-ggm8c_3ff1c407-8b92-4657-82ac-eb65af7d9bbe/main/0.log"
Oct 30 01:38:00.812075 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:38:00.812040    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-s7t4h_770ca287-b172-4664-9e09-f024674f5c9c/main/0.log"
Oct 30 01:38:01.184708 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:38:01.184682    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-g95b2_68d0836f-44b4-45d7-a37f-bd20ce93a93f/main/0.log"
Oct 30 01:38:26.509643 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:38:26.509599    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:38:34.017229 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:38:34.017149    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6b5b68ff8f-jrgsq_a06f7a71-cfbc-4a87-a668-bb3fa21ee2ef/manager/0.log"
Oct 30 01:38:34.355734 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:38:34.355640    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6b5b68ff8f-jrgsq_a06f7a71-cfbc-4a87-a668-bb3fa21ee2ef/manager/0.log"
Oct 30 01:38:35.022972 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:38:35.022945    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-l4gst_9217cef0-6bf2-41d5-b303-4c4b78dcb7f0/payload-processing/0.log"
Oct 30 01:38:35.217460 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:38:35.217427    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-l4gst_9217cef0-6bf2-41d5-b303-4c4b78dcb7f0/payload-processing/0.log"
Oct 30 01:38:56.510091 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:38:56.510046    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:39:17.428373 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:17.428339    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6b5b68ff8f-jrgsq_a06f7a71-cfbc-4a87-a668-bb3fa21ee2ef/manager/0.log"
Oct 30 01:39:17.790874 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:17.790771    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6b5b68ff8f-jrgsq_a06f7a71-cfbc-4a87-a668-bb3fa21ee2ef/manager/0.log"
Oct 30 01:39:18.535243 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:18.535212    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-l4gst_9217cef0-6bf2-41d5-b303-4c4b78dcb7f0/payload-processing/0.log"
Oct 30 01:39:18.732223 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:18.732198    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-l4gst_9217cef0-6bf2-41d5-b303-4c4b78dcb7f0/payload-processing/0.log"
Oct 30 01:39:19.074977 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:19.074942    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dhfxw/must-gather-d7kxl"]
Oct 30 01:39:19.075453 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:19.075438    4393 cpu_manager.go:414] "containerMap: removing container" podUID="29c80ca8-5215-41d3-9c4c-a4744be8b019" containerName="maas-api"
Oct 30 01:39:19.075513 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:19.075454    4393 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29c80ca8-5215-41d3-9c4c-a4744be8b019" containerName="maas-api"
Oct 30 01:39:19.075552 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:19.075544    4393 memory_manager.go:365] "RemoveStaleState removing state" podUID="29c80ca8-5215-41d3-9c4c-a4744be8b019" containerName="maas-api"
Oct 30 01:39:19.078920 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:19.078901    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dhfxw/must-gather-d7kxl"
Oct 30 01:39:19.104701 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:19.104674    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dhfxw\"/\"kube-root-ca.crt\""
Oct 30 01:39:19.104878 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:19.104754    4393 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dhfxw\"/\"default-dockercfg-tt658\""
Oct 30 01:39:19.104878 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:19.104764    4393 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dhfxw\"/\"openshift-service-ca.crt\""
Oct 30 01:39:19.129189 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:19.129164    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkgsg\" (UniqueName: \"kubernetes.io/projected/5665b4af-fd6f-4cb0-a52d-993b74452e7a-kube-api-access-nkgsg\") pod \"must-gather-d7kxl\" (UID: \"5665b4af-fd6f-4cb0-a52d-993b74452e7a\") " pod="openshift-must-gather-dhfxw/must-gather-d7kxl"
Oct 30 01:39:19.129319 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:19.129228    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5665b4af-fd6f-4cb0-a52d-993b74452e7a-must-gather-output\") pod \"must-gather-d7kxl\" (UID: \"5665b4af-fd6f-4cb0-a52d-993b74452e7a\") " pod="openshift-must-gather-dhfxw/must-gather-d7kxl"
Oct 30 01:39:19.142617 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:19.142589    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dhfxw/must-gather-d7kxl"]
Oct 30 01:39:19.230009 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:19.229978    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5665b4af-fd6f-4cb0-a52d-993b74452e7a-must-gather-output\") pod \"must-gather-d7kxl\" (UID: \"5665b4af-fd6f-4cb0-a52d-993b74452e7a\") " pod="openshift-must-gather-dhfxw/must-gather-d7kxl"
Oct 30 01:39:19.230164 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:19.230081    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkgsg\" (UniqueName: \"kubernetes.io/projected/5665b4af-fd6f-4cb0-a52d-993b74452e7a-kube-api-access-nkgsg\") pod \"must-gather-d7kxl\" (UID: \"5665b4af-fd6f-4cb0-a52d-993b74452e7a\") " pod="openshift-must-gather-dhfxw/must-gather-d7kxl"
Oct 30 01:39:19.230325 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:19.230306    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5665b4af-fd6f-4cb0-a52d-993b74452e7a-must-gather-output\") pod \"must-gather-d7kxl\" (UID: \"5665b4af-fd6f-4cb0-a52d-993b74452e7a\") " pod="openshift-must-gather-dhfxw/must-gather-d7kxl"
Oct 30 01:39:19.279833 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:19.279801    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkgsg\" (UniqueName: \"kubernetes.io/projected/5665b4af-fd6f-4cb0-a52d-993b74452e7a-kube-api-access-nkgsg\") pod \"must-gather-d7kxl\" (UID: \"5665b4af-fd6f-4cb0-a52d-993b74452e7a\") " pod="openshift-must-gather-dhfxw/must-gather-d7kxl"
Oct 30 01:39:19.392748 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:19.392714    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dhfxw/must-gather-d7kxl"
Oct 30 01:39:19.701600 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:19.701574    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dhfxw/must-gather-d7kxl"]
Oct 30 01:39:19.702646 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:39:19.702616    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5665b4af_fd6f_4cb0_a52d_993b74452e7a.slice/crio-3d98d749e45c801346a523dad05d4d97898f9d02a506721ee698bd0b67cb145f WatchSource:0}: Error finding container 3d98d749e45c801346a523dad05d4d97898f9d02a506721ee698bd0b67cb145f: Status 404 returned error can't find the container with id 3d98d749e45c801346a523dad05d4d97898f9d02a506721ee698bd0b67cb145f
Oct 30 01:39:19.713076 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:19.713050    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhfxw/must-gather-d7kxl" event={"ID":"5665b4af-fd6f-4cb0-a52d-993b74452e7a","Type":"ContainerStarted","Data":"3d98d749e45c801346a523dad05d4d97898f9d02a506721ee698bd0b67cb145f"}
Oct 30 01:39:20.721626 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:20.721577    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhfxw/must-gather-d7kxl" event={"ID":"5665b4af-fd6f-4cb0-a52d-993b74452e7a","Type":"ContainerStarted","Data":"6f0b9b74e18fcc31d5022bf5cc89ba9cffea9ff19771e39399fac78ebf162dbd"}
Oct 30 01:39:20.722191 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:20.721636    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhfxw/must-gather-d7kxl" event={"ID":"5665b4af-fd6f-4cb0-a52d-993b74452e7a","Type":"ContainerStarted","Data":"0090ca74bd5139165509cf272644a48ddd22623a14aca09306949fe70a89f142"}
Oct 30 01:39:20.764289 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:20.764236    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-dhfxw/must-gather-d7kxl" podStartSLOduration=1.14198588 podStartE2EDuration="1.764223099s" totalImagesPullingTime="622.237219ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:39:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:39:20.760943028 +0000 UTC m=+4051.148619416" watchObservedRunningTime="2026-10-30 01:39:20.764223099 +0000 UTC m=+4051.151899488"
Oct 30 01:39:23.272548 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:23.272517    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-f27vk_3b0df054-0749-4417-bb8b-b12b3353d25e/global-pull-secret-syncer/0.log"
Oct 30 01:39:24.490797 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:24.486681    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6wzd9_4f7ca654-edeb-4457-9239-8da741b5dafd/konnectivity-agent/0.log"
Oct 30 01:39:25.004800 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:24.989964    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-56.ec2.internal_21539217511a9b702eb77d59ab0b339d/haproxy/0.log"
Oct 30 01:39:25.013698 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:25.007760    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-56.ec2.internal_21539217511a9b702eb77d59ab0b339d/haproxy/1.log"
Oct 30 01:39:26.615317 ip-10-0-135-56 kubenswrapper[4393]: E1030 01:39:26.615116    4393 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jbmfs" containerName="extractor"
Oct 30 01:39:31.256825 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:31.255077    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-kvgjz_7bd77191-8d97-498a-978b-0765096177a1/manager/0.log"
Oct 30 01:39:31.385087 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:31.385058    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-xgjkr_4c9fdd3e-ff82-404c-a637-1b18a139938d/manager/0.log"
Oct 30 01:39:31.687803 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:31.687645    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-4zr2r_c2d79a11-d151-4368-9469-2034d93cd39d/manager/0.log"
Oct 30 01:39:33.381135 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:33.381094    4393 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dhfxw/perf-node-gather-daemonset-6m26l"]
Oct 30 01:39:33.385019 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:33.385000    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-6m26l"
Oct 30 01:39:33.405380 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:33.405346    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dhfxw/perf-node-gather-daemonset-6m26l"]
Oct 30 01:39:33.497377 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:33.497331    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b695bde4-7eb8-4451-8166-bd6008393dd1-proc\") pod \"perf-node-gather-daemonset-6m26l\" (UID: \"b695bde4-7eb8-4451-8166-bd6008393dd1\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-6m26l"
Oct 30 01:39:33.497608 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:33.497471    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8l2p\" (UniqueName: \"kubernetes.io/projected/b695bde4-7eb8-4451-8166-bd6008393dd1-kube-api-access-s8l2p\") pod \"perf-node-gather-daemonset-6m26l\" (UID: \"b695bde4-7eb8-4451-8166-bd6008393dd1\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-6m26l"
Oct 30 01:39:33.497608 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:33.497509    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b695bde4-7eb8-4451-8166-bd6008393dd1-lib-modules\") pod \"perf-node-gather-daemonset-6m26l\" (UID: \"b695bde4-7eb8-4451-8166-bd6008393dd1\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-6m26l"
Oct 30 01:39:33.497608 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:33.497556    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b695bde4-7eb8-4451-8166-bd6008393dd1-podres\") pod \"perf-node-gather-daemonset-6m26l\" (UID: \"b695bde4-7eb8-4451-8166-bd6008393dd1\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-6m26l"
Oct 30 01:39:33.497828 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:33.497615    4393 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b695bde4-7eb8-4451-8166-bd6008393dd1-sys\") pod \"perf-node-gather-daemonset-6m26l\" (UID: \"b695bde4-7eb8-4451-8166-bd6008393dd1\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-6m26l"
Oct 30 01:39:33.599078 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:33.599031    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b695bde4-7eb8-4451-8166-bd6008393dd1-podres\") pod \"perf-node-gather-daemonset-6m26l\" (UID: \"b695bde4-7eb8-4451-8166-bd6008393dd1\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-6m26l"
Oct 30 01:39:33.599309 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:33.599098    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b695bde4-7eb8-4451-8166-bd6008393dd1-sys\") pod \"perf-node-gather-daemonset-6m26l\" (UID: \"b695bde4-7eb8-4451-8166-bd6008393dd1\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-6m26l"
Oct 30 01:39:33.599309 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:33.599158    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b695bde4-7eb8-4451-8166-bd6008393dd1-proc\") pod \"perf-node-gather-daemonset-6m26l\" (UID: \"b695bde4-7eb8-4451-8166-bd6008393dd1\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-6m26l"
Oct 30 01:39:33.599309 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:33.599226    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8l2p\" (UniqueName: \"kubernetes.io/projected/b695bde4-7eb8-4451-8166-bd6008393dd1-kube-api-access-s8l2p\") pod \"perf-node-gather-daemonset-6m26l\" (UID: \"b695bde4-7eb8-4451-8166-bd6008393dd1\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-6m26l"
Oct 30 01:39:33.599309 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:33.599232    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b695bde4-7eb8-4451-8166-bd6008393dd1-podres\") pod \"perf-node-gather-daemonset-6m26l\" (UID: \"b695bde4-7eb8-4451-8166-bd6008393dd1\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-6m26l"
Oct 30 01:39:33.599309 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:33.599234    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b695bde4-7eb8-4451-8166-bd6008393dd1-sys\") pod \"perf-node-gather-daemonset-6m26l\" (UID: \"b695bde4-7eb8-4451-8166-bd6008393dd1\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-6m26l"
Oct 30 01:39:33.599309 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:33.599290    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b695bde4-7eb8-4451-8166-bd6008393dd1-proc\") pod \"perf-node-gather-daemonset-6m26l\" (UID: \"b695bde4-7eb8-4451-8166-bd6008393dd1\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-6m26l"
Oct 30 01:39:33.599670 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:33.599327    4393 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b695bde4-7eb8-4451-8166-bd6008393dd1-lib-modules\") pod \"perf-node-gather-daemonset-6m26l\" (UID: \"b695bde4-7eb8-4451-8166-bd6008393dd1\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-6m26l"
Oct 30 01:39:33.599670 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:33.599443    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b695bde4-7eb8-4451-8166-bd6008393dd1-lib-modules\") pod \"perf-node-gather-daemonset-6m26l\" (UID: \"b695bde4-7eb8-4451-8166-bd6008393dd1\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-6m26l"
Oct 30 01:39:33.627512 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:33.627475    4393 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8l2p\" (UniqueName: \"kubernetes.io/projected/b695bde4-7eb8-4451-8166-bd6008393dd1-kube-api-access-s8l2p\") pod \"perf-node-gather-daemonset-6m26l\" (UID: \"b695bde4-7eb8-4451-8166-bd6008393dd1\") " pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-6m26l"
Oct 30 01:39:33.632666 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:33.632580    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_89bbffef-b6f8-44df-9752-b1c163a9abea/alertmanager/0.log"
Oct 30 01:39:33.699980 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:33.699948    4393 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-6m26l"
Oct 30 01:39:33.761395 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:33.760763    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_89bbffef-b6f8-44df-9752-b1c163a9abea/config-reloader/0.log"
Oct 30 01:39:33.880088 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:33.880051    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_89bbffef-b6f8-44df-9752-b1c163a9abea/kube-rbac-proxy-web/0.log"
Oct 30 01:39:34.009481 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:34.009402    4393 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dhfxw/perf-node-gather-daemonset-6m26l"]
Oct 30 01:39:34.010486 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:34.010458    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_89bbffef-b6f8-44df-9752-b1c163a9abea/kube-rbac-proxy/0.log"
Oct 30 01:39:34.021962 ip-10-0-135-56 kubenswrapper[4393]: W1030 01:39:34.015895    4393 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb695bde4_7eb8_4451_8166_bd6008393dd1.slice/crio-7231003ae9ebb5cd11fe610c0d7e53014bb765a3c1be913bc96eac8ad92c8069 WatchSource:0}: Error finding container 7231003ae9ebb5cd11fe610c0d7e53014bb765a3c1be913bc96eac8ad92c8069: Status 404 returned error can't find the container with id 7231003ae9ebb5cd11fe610c0d7e53014bb765a3c1be913bc96eac8ad92c8069
Oct 30 01:39:34.131069 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:34.131043    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_89bbffef-b6f8-44df-9752-b1c163a9abea/kube-rbac-proxy-metric/0.log"
Oct 30 01:39:34.258851 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:34.258810    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_89bbffef-b6f8-44df-9752-b1c163a9abea/prom-label-proxy/0.log"
Oct 30 01:39:34.382952 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:34.382921    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_89bbffef-b6f8-44df-9752-b1c163a9abea/init-config-reloader/0.log"
Oct 30 01:39:34.847815 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:34.847719    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-6m26l" event={"ID":"b695bde4-7eb8-4451-8166-bd6008393dd1","Type":"ContainerStarted","Data":"a84e98976096455c27c840656f929300ef937d74738169ddae92f508dc244584"}
Oct 30 01:39:34.847978 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:34.847844    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-6m26l"
Oct 30 01:39:34.847978 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:34.847864    4393 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-6m26l" event={"ID":"b695bde4-7eb8-4451-8166-bd6008393dd1","Type":"ContainerStarted","Data":"7231003ae9ebb5cd11fe610c0d7e53014bb765a3c1be913bc96eac8ad92c8069"}
Oct 30 01:39:34.889245 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:34.889179    4393 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-6m26l" podStartSLOduration=1.8891608720000002 podStartE2EDuration="1.889160872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:39:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:39:34.888349798 +0000 UTC m=+4065.276026186" watchObservedRunningTime="2026-10-30 01:39:34.889160872 +0000 UTC m=+4065.276837262"
Oct 30 01:39:35.243251 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:35.243201    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-67pwz_8e23449e-a558-4347-bed9-e631437e9c73/node-exporter/0.log"
Oct 30 01:39:35.363902 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:35.363868    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-67pwz_8e23449e-a558-4347-bed9-e631437e9c73/kube-rbac-proxy/0.log"
Oct 30 01:39:35.491050 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:35.491019    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-67pwz_8e23449e-a558-4347-bed9-e631437e9c73/init-textfile/0.log"
Oct 30 01:39:37.782317 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:37.782272    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-p5zpd_7258c681-0692-4203-b3da-a3bab6b298dd/prometheus-operator-admission-webhook/0.log"
Oct 30 01:39:37.910755 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:37.910707    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-766c9f88f-jvcvh_2fa1bcf9-0df3-4978-a8a4-493c6b066681/telemeter-client/0.log"
Oct 30 01:39:38.034080 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:38.033985    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-766c9f88f-jvcvh_2fa1bcf9-0df3-4978-a8a4-493c6b066681/reload/0.log"
Oct 30 01:39:38.157486 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:38.157459    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-766c9f88f-jvcvh_2fa1bcf9-0df3-4978-a8a4-493c6b066681/kube-rbac-proxy/0.log"
Oct 30 01:39:38.290454 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:38.290356    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-648d8c566c-nw7rf_4af88d81-38c0-4edf-be60-d8c27b330779/thanos-query/0.log"
Oct 30 01:39:38.416080 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:38.416045    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-648d8c566c-nw7rf_4af88d81-38c0-4edf-be60-d8c27b330779/kube-rbac-proxy-web/0.log"
Oct 30 01:39:38.543462 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:38.543370    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-648d8c566c-nw7rf_4af88d81-38c0-4edf-be60-d8c27b330779/kube-rbac-proxy/0.log"
Oct 30 01:39:38.672212 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:38.672184    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-648d8c566c-nw7rf_4af88d81-38c0-4edf-be60-d8c27b330779/prom-label-proxy/0.log"
Oct 30 01:39:38.795349 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:38.795269    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-648d8c566c-nw7rf_4af88d81-38c0-4edf-be60-d8c27b330779/kube-rbac-proxy-rules/0.log"
Oct 30 01:39:38.919796 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:38.919750    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-648d8c566c-nw7rf_4af88d81-38c0-4edf-be60-d8c27b330779/kube-rbac-proxy-metrics/0.log"
Oct 30 01:39:40.865061 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:40.865033    4393 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dhfxw/perf-node-gather-daemonset-6m26l"
Oct 30 01:39:42.685703 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:42.685671    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-77577bccf6-6vmfm_e5aedfc1-943a-43a7-bd61-a58892dd8428/console/0.log"
Oct 30 01:39:42.842321 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:42.842287    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-r7bf7_4a3d031e-b63e-4201-afb9-ec659b67ac1e/download-server/0.log"
Oct 30 01:39:46.633145 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:46.633103    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw_d685e131-7f49-4c0d-870f-1d1a51bd1b74/istio-proxy/0.log"
Oct 30 01:39:46.791294 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:46.791254    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9ml2g_b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f/dns/0.log"
Oct 30 01:39:46.913773 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:46.913685    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9ml2g_b3aa0462-2e3d-4df7-9632-2f1c28ad6d6f/kube-rbac-proxy/0.log"
Oct 30 01:39:47.448114 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:47.448060    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fqtzs_05a614f5-3112-4216-b894-68bf3945458c/dns-node-resolver/0.log"
Oct 30 01:39:48.797683 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:48.797632    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b7df967f4-5hvrq_e1c4b01c-5605-474b-8e98-91f81fe6873d/kube-auth-proxy/0.log"
Oct 30 01:39:48.878541 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:48.878494    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7d899fbfd7-2j85r_21e682d8-71fc-4a64-9eda-1b5ed1788fde/registry/0.log"
Oct 30 01:39:48.971243 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:48.971214    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-26zqk_58e0c954-cdcc-4ad1-bbf7-a2197816e842/istio-proxy/0.log"
Oct 30 01:39:49.135414 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:49.135380    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-97m56_d4da6381-6d65-4f27-9f7f-907dd8b5da23/istio-proxy/0.log"
Oct 30 01:39:49.285001 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:49.284969    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-cngvb_f3bd5875-a930-4985-b820-efe272d30763/istio-proxy/0.log"
Oct 30 01:39:49.287022 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:49.286995    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7d9897b54c-cngvb_f3bd5875-a930-4985-b820-efe272d30763/istio-proxy/1.log"
Oct 30 01:39:49.343564 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:49.343528    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9hslh_30e43c48-92a7-4c68-887c-9aaf7d6d1e1b/node-ca/0.log"
Oct 30 01:39:49.782388 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:49.782354    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-l4gst_9217cef0-6bf2-41d5-b303-4c4b78dcb7f0/payload-processing/0.log"
Oct 30 01:39:51.327177 ip-10-0-135-56 kubenswrapper[4393]: I1030 01:39:51.327148    4393 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-858f55f4d9kq5rw_d685e131-7f49-4c0d-870f-1d1a51bd1b74/istio-proxy/0.log"